Regularization (Penalizing Complexity)¶

Why It Matters¶

Without regularization, optimization naturally produces the most complex model that fits your data - which is almost always overfit. Humans do this too: we construct elaborate theories that perfectly explain our limited experience but fail on new data. Regularization is formalized Occam's Razor with teeth.

The Key Move¶

When building any model or explanation, explicitly add a complexity penalty. For every additional parameter, variable, or exception you add, demand proportionally stronger evidence. Ask: "Does this complexity pay for itself in predictive improvement, or am I just explaining noise?"

Classic Application¶

Ridge and LASSO regression. Without regularization, adding variables always improves training fit, encouraging you to include every possible predictor. Regularization shrinks coefficients toward zero, effectively removing weak predictors. Result: simpler models that generalize better.

Surprising Application¶

Conspiracy theories and motivated reasoning. Complex explanations can fit any data if you add enough epicycles. Regularization as a cognitive discipline: penalize explanations that require many coordinated secret actors or unlikely coincidences. The prior probability penalty on complexity often outweighs the marginal improvement in fitting observations.

Failure Modes¶

Over-regularization produces models too simple to capture real patterns - you need some complexity. Choosing the penalty strength is itself a modeling choice requiring validation. Domain knowledge sometimes justifies complexity that looks like overfitting - not all complicated explanations are wrong.

Go Deeper¶

Hastie et al., Elements of Statistical Learning, Chapter 3.4; Murphy, Machine Learning: A Probabilistic Perspective, Chapter 13