Spark + dbt · Databricks · Prediction Markets

World Cup prediction market pipeline

I built this as two parallel pipelines over Manifold Markets' public API and Polymarket's API, testing calibration and the favorite-longshot bias in 2026 World Cup markets. One runs Spark, dbt, and DuckDB or Postgres as a Kubernetes batch job; the other runs the same logic on Databricks with Delta Live Tables and Unity Catalog.

Polymarket trades on real money. I added it to test whether calibration actually differs between a real-money market and Manifold's own play-money Mana, since Mana can be bought with cash but never cashed back out.

I built both paths on purpose. Spark, dbt, Kubernetes, and Databricks aren't what I use day to day, and I wanted real hands-on time with them.

Code on GitHub

Calibration plot of Manifold World Cup 2026 markets: predicted probability at resolution versus actual resolution rate, closely tracking the perfect-calibration line — Market calibration: predicted probability vs. actual resolution rate

← Back to projects