Evidence Strength Calibration¶

Why It Matters¶

Humans are notoriously miscalibrated - we're overconfident, we misestimate likelihood ratios, we update too much on weak evidence and too little on strong evidence. But these biases are correctable if measured. Evidence strength calibration provides feedback: it reveals your personal bias patterns, allowing you to develop correction factors. "When I think evidence is strong, I'm actually right 60% of the time - I should be less certain."

The Key Move¶

When evaluating evidence, make explicit probabilistic predictions. Write them down with timestamps and the reasoning that led to them. After outcomes resolve, score yourself: what fraction of your "90% confident" predictions came true? Did your "strong evidence" cases confirm more often than your "weak evidence" cases? Calculate your calibration curve. If you're overconfident (90% predictions only 70% accurate), develop a correction habit - discount your confidence. If you're underconfident, trust your evidence assessments more.

Classic Application¶

Weather forecasting. Meteorologists say "70% chance of rain" for thousands of forecasts. Calibration check: on days they said 70%, did it rain 70% of the time? Well-calibrated forecasters match predictions to frequencies. Poorly calibrated forecasters might say "70%" but actually mean 50% (overconfident) or 85% (underconfident). Tracking calibration lets them correct systematic biases.

Surprising Application¶

Personal relationship predictions. You make implicit forecasts about people: "They'll probably be late" (80% confidence), "They'll love this gift" (60% confidence). Start tracking these. After a year, check calibration: were your 80% predictions right 80% of the time? You might discover you're systematically overconfident about punctuality (your 80% predictions only 50% accurate - you're bad at predicting lateness) but well-calibrated on gift preferences. This reveals where your social models are accurate versus biased.

Failure Modes¶

Insufficient sample size - calibration requires many predictions; checking after 5 cases is meaningless. Reference class sensitivity - calibration depends on the domain; being calibrated on weather doesn't mean you're calibrated on politics. Feedback delays - if outcomes take years to resolve, calibration learning is very slow. Selection bias - you only track predictions you remember making, which correlates with confidence and outcome, biasing calibration estimates.

Go Deeper¶

Tetlock & Gardner, Superforecasting, Chapter 2 on calibration; Lichtenstein et al., "Calibration of Probabilities: The State of the Art to 1980" (1982).