Confidence Calibration¶

Why It Matters¶

Analysts naturally provide point estimates ("X will happen") without quantifying uncertainty. Consumers misinterpret qualitative language ("likely," "probable") differently. Poor calibration means overconfidence in wrong judgments and underconfidence in correct ones. Explicit probability forces precise communication and enables learning from feedback.

The Key Move¶

For each judgment, assign a probability: "I assess 70% probability that X will occur." Use the full range (avoid clustering at 50% or 95%). Distinguish confidence in judgment from importance of judgment. Track your judgments and outcomes. Calculate calibration: when you say 70%, are you right 70% of the time? If you're right 90% of the time you say 70%, you're underconfident. If you're right 50%, you're overconfident. Adjust based on track record.

Classic Application¶

Intelligence Community Directive 203 (2015) mandated probability estimates for intelligence assessments. Rather than "Iraq likely has WMD," analysts must quantify: "We assess with moderate confidence (roughly 50-60% probability) that Iraq has reconstituted weapons programs." This forces explicit uncertainty communication and enables post-assessment learning.

Surprising Application¶

Medical diagnosis. Rather than "I think it's appendicitis," physicians trained in calibration say: "I assess 80% probability of appendicitis given symptoms, exam findings, and lab results." This changes treatment decisions (80% warrants surgery; 40% warrants imaging first) and enables learning (did the 80% cases actually have appendicitis? If only 60%, the physician is overconfident and should adjust).

Failure Modes¶

Using probabilities as rhetorical emphasis rather than genuine uncertainty (saying 95% when you mean "very confident" but haven't actually assessed probability). Not tracking outcomes (you can't calibrate without feedback). Binary thinking (everything is either 5% or 95%, nothing in middle). Confusing confidence with probability (feeling certain doesn't mean high probability). Not updating probabilities as new information arrives (Bayesian updating required).

Go Deeper¶

Tetlock and Gardner, "Superforecasting" on calibration training. Mandel, "Improving Intelligence Analysis: Methodological and Conceptual Foundations" Chapter 5. Friedman and Zeckhauser, "Assessing Uncertainty in Intelligence" (Intelligence and National Security, 2012).