The Curse of Dimensionality¶

Why It Matters¶

Humans evolved to reason in 3D space and assume that intuitions scale. They don't. Adding more measurements seems like it should always help, but in high dimensions, it usually hurts unless you have proportionally more data. This explains why "measure everything" strategies often fail.

The Key Move¶

When adding new features or variables to any analysis, ask: "Do I have enough data to support this dimensionality?" A rough heuristic: you need exponentially more samples as dimensions increase. If data is scarce, aggressive dimensionality reduction (feature selection, aggregation, or learned embeddings) isn't optional - it's necessary for learning to work at all.

Classic Application¶

K-nearest neighbors classification. In 2D with 100 samples, you get good coverage. In 100D with 100 samples, every point is approximately equidistant from every other point - "nearest" neighbors are nearly as far as the farthest ones. The algorithm becomes random guessing.

Surprising Application¶

Decision paralysis and option overload. Choosing from a few options is tractable; choosing from many dimensions simultaneously (career, location, salary, culture, growth, commute, colleagues...) creates a high-dimensional space where you can't sample densely. Solution: reduce dimensions by identifying composite factors or accepting that you can't optimize everything simultaneously.

Failure Modes¶

Assumes features are independent - if they're correlated, effective dimensionality is lower. Smooth low-dimensional manifold hypothesis - sometimes high-dimensional data lies near a low-dimensional surface, escaping the curse. Over-pruning dimensions loses critical information.

Go Deeper¶

Hastie et al., Elements of Statistical Learning, Section 2.5; Bellman (1961), Adaptive Control Processes (original source)