Skip to content

Learning Curves (Sample Efficiency Diagnosis)

Why It Matters

When models underperform, people default to "add more data" or "try a more complex model" without diagnosing the actual bottleneck. Learning curves make the tradeoffs visible: flat curves mean more data won't help; converging curves mean you're near optimal; diverging train/test curves mean overfitting.

The Key Move

Plot performance (y-axis) against training examples (x-axis) for both training and test sets. Diagnosis:

  1. Large train-test gap = overfitting, try regularization or simpler models.
  2. Both plateaued at poor performance = underfitting, try more complex models or better features.
  3. Test improving with no plateau = collect more data.
  4. Both near-perfect = task is solved.

Classic Application

Speech recognition. Early systems showed test performance steadily improving with data and no plateau, indicating data was the bottleneck. This justified massive data collection efforts that eventually solved the problem. Without learning curves, teams might have pivoted to different algorithms prematurely.

Surprising Application

Skill acquisition and deliberate practice. Track performance against practice hours. If learning plateaus despite practice (flat curve), the bottleneck isn't practice quantity - you need better feedback, different training methods, or fundamentals you're missing. If learning continues linearly, keep doing what you're doing.

Failure Modes

Requires ability to vary training size - can't do this with fixed small datasets. Non-monotonic curves from high variance - need multiple runs to see true trends. Extrapolation uncertainty - curves can have inflection points, so early data doesn't reliably predict final performance.

Go Deeper

Perlich, Provost & Simonoff (2003), "Tree Induction vs. Logistic Regression: A Learning-Curve Analysis," JMLR; Hestness et al. (2017), "Deep Learning Scaling is Predictable, Empirically," arXiv