Model Skill Assessment¶

Why It Matters¶

Humans are impressed by complex models and can mistake sophistication for usefulness. A model with 70% accuracy sounds good until you realize random guessing would get 50% and climatology gets 75%. Skill assessment reveals whether your fancy analysis actually adds value or just creates the appearance of rigor. It prevents wasting effort on models that are worse than simple heuristics.

The Key Move¶

For any model or prediction method, identify the appropriate naive baseline (what's the dumbest thing that might work?). Measure your model's performance and the baseline's performance on the same data. Calculate the skill score: (Model - Baseline) / (Perfect - Baseline). If skill score is negative, your model is worse than doing nothing. If it's near zero, your complexity adds no value. Only trust models with substantial positive skill, and track whether skill degrades over time.

Classic Application¶

Weather forecast verification. The naive persistence forecast (tomorrow's weather = today's weather) is hard to beat at short range. Climatology (average weather for this date) is the baseline for longer ranges. Modern forecasts have skill scores of 0.6-0.8 at short range, meaning they capture 60-80% of the theoretically possible improvement over climatology. Below 0.2, the forecast is barely useful.

Surprising Application¶

Hiring decisions. Many companies use elaborate interview processes without measuring if they outperform simple baselines: hiring randomly from qualified applicants, using only resume screening, or asking just one or two structured questions. If your 6-round interview process has the same outcome as 2 structured interviews, the extra 4 rounds have zero skill. Measure actual job performance against hiring process complexity to see if you're beating the baseline.

Failure Modes¶

Wrong baseline - using "random chance" when persistence or climatology is the real alternative. Overfitting - measuring skill on training data rather than held-out test cases. Skill conflation - confusing statistical significance with practical importance (a "significant" improvement of 0.5% may have near-zero skill). Ignoring cost - a high-skill model that costs 100x more than a medium-skill baseline may not be worth it. Cherry-picking - reporting skill only on cases where the model works.

Go Deeper¶

Wilks, Statistical Methods in the Atmospheric Sciences, Chapter 8; Murphy & Epstein, "Skill Scores and Correlation Coefficients in Model Verification," Monthly Weather Review (1989).

Screening - Information Economics: Reasoning Under Asymmetry and Uncertainty