Cross-Validation (Robust Performance Estimation)¶

Why It Matters¶

Single train-test splits have high variance - you might get lucky or unlucky with which data lands in test. This creates false confidence or false despair. Cross-validation reduces this variance by averaging over multiple tests, giving you honest uncertainty estimates.

The Key Move¶

Divide data into k folds (typically 5-10). Train on k-1 folds, test on the held-out fold, repeat k times with each fold serving as test once. Report mean and standard deviation of performance. Use this for model selection, hyperparameter tuning, or any decision that depends on performance estimates. Never use cross-validation performance to make final claims - that's still a form of overfitting.

Classic Application¶

Model selection with limited data. You have 1000 examples and three candidate algorithms. Single train-test split (700/300) gives noisy estimates. 5-fold cross-validation uses all data for both training and testing (in rotation), giving much more reliable algorithm rankings.

Surprising Application¶

A/B testing and experimental design in small organizations. You can't run endless experiments with tiny user bases. Cross-validation principle: instead of one long test, run multiple short tests with different user segments, then aggregate. Averages out segment-specific noise while using all available data.

Failure Modes¶

Temporal data - random splits break time dependencies; use time-series splits instead. Grouped data - random splits can leak information between groups (e.g., multiple samples from same patient); use grouped cross-validation. Computational cost - k-fold requires training k models. Still overfit through hyperparameter search if not nested properly.

Go Deeper¶

Hastie et al., Elements of Statistical Learning, Section 7.10; Arlot & Celisse (2010), "A Survey of Cross-Validation Procedures for Model Selection," Statistics Surveys