Verification and Calibration Loops¶

Why It Matters¶

Without verification, forecasts drift into wishful thinking or conventional wisdom rather than reality. Humans have terrible intuitive calibration - we systematically over-weight recent events, anchor on our own predictions, and avoid updating when we're wrong. Formal verification forces confrontation with reality and enables actual learning rather than narrative adjustment.

The Key Move¶

Archive every forecast with specifics: what was predicted, with what confidence, for what time and location. When the outcome is known, compare: what happened, what was predicted, what was the error? Aggregate verification statistics: bias (systematic over/under prediction), skill (improvement over baseline), calibration (do 70% forecasts verify 70% of the time?), sharpness (how narrow are the confidence intervals?). Feed this analysis back into the forecasting process: adjust biased models, improve low-skill components, recalibrate confidence.

Classic Application¶

NWS forecast verification. Every precipitation forecast is archived and compared to observations. Statistics are computed: probability of detection, false alarm rate, bias, skill scores. These metrics identify systematic errors (models over-predict warm-sector rainfall, under-predict lake-effect snow) which drive model development priorities. Verification is continuous and public - forecast quality improves because errors are systematically tracked and addressed.

Surprising Application¶

Personal decision-making. Keep a decision journal: major decisions, your prediction of outcome, your confidence level, the actual outcome. After 20-30 decisions, analyze: are your "80% confident" predictions right 80% of the time? Do you over-predict positive outcomes (optimism bias)? Do certain types of decisions have lower skill? Use this to calibrate future confidence ("I'm usually overconfident about social outcomes, so my 80% is really 60%") and identify decision types where you should defer to others or use formal methods.

Failure Modes¶

Cherry-picking verification cases - only checking predictions that worked or only in favorable conditions. Delayed verification - waiting so long that you can't remember your reasoning or adjust processes. Overfitting to recent errors - overcorrecting based on latest failures rather than systematic patterns. Missing decomposition - treating all errors as equal rather than distinguishing bias, random error, and rare events. Verification without action - measuring everything but changing nothing.

Go Deeper¶

Wilks, Statistical Methods in the Atmospheric Sciences, Chapter 8; Tetlock & Gardner, Superforecasting, Chapter 2.