Posterior Predictive Checking¶

Why It Matters¶

It's easy to feel satisfied after a Bayesian update: you've incorporated the evidence, you have a new posterior, you're done. But this is false comfort if your model is wrong. Posterior predictive checking imposes discipline: use your updated beliefs to generate testable predictions, then check them against reality. Discrepancies reveal model misspecification, not just parameter uncertainty. This closes the feedback loop between belief and reality.

The Key Move¶

After updating to a posterior belief, ask: "If my posterior is correct, what should I expect to see next?" Generate specific predictions with probabilities. When new data arrives, compare to predictions. If data falls in the predicted range, your model is validated (for now). If data is surprising given your posterior, don't just update parameters - question your model structure. What assumptions might be wrong? Do you need a richer hypothesis space?

Classic Application¶

Election forecasting models. After updating on polls to generate posterior probability of candidate victory, the model should predict poll results in new states/demographics. If actual polls consistently fall outside predicted ranges, the model is misspecified - maybe it's underestimating uncertainty, missing a systematic bias, or using wrong priors on voter groups. Posterior predictive checking catches these failures before election day.

Surprising Application¶

Personal habit formation. You've updated on evidence to believe you have 70% probability of maintaining an exercise habit (posterior). Posterior predictive check: "If I truly have 70% success probability, I should exercise about 5 of the next 7 days." After a week, compare. If you only exercised 2 days, your model is wrong - maybe your probability estimate was too optimistic, or maybe you're not accounting for motivation dynamics. Revise model or gather different evidence, don't just update the number.

Failure Modes¶

Cherry-picking predictions - generating only predictions you expect to pass. Vague predictions - "something good should happen" isn't checkable. Ignoring model class uncertainty - posterior predictive checking assumes your model structure is right; it won't catch errors that all models in your class share. Overreacting to single failures - prediction checking requires multiple tests; one surprise might be noise.

Go Deeper¶

Gelman et al., Bayesian Data Analysis, Chapter 6 on posterior predictive checking; Box, "Sampling and Bayes' Inference in Scientific Modelling and Robustness" (1980).

Surprise - Military Strategy & Strategic Thinking: A Map of the Territory