Ensemble Methods (Wisdom of Crowds)¶

Why It Matters¶

The search for "the best model" is usually misguided. Different models make different assumptions and capture different patterns. Combining diverse models typically outperforms any individual model because errors are partially independent. This challenges the single-truth intuition in favor of triangulation.

The Key Move¶

When making predictions or decisions, deliberately construct multiple independent models or perspectives. Key requirement: they must be genuinely diverse (different training data, different algorithms, or different feature sets). Average or vote across them. The benefit comes from diversity, not just quantity - ten variations of the same approach won't help.

Classic Application¶

Random forests. Instead of one decision tree, grow hundreds with different random subsets of data and features. Each tree is mediocre and overfit, but their errors are partially independent. Averaging predictions dramatically improves generalization - often the best off-the-shelf method for structured data.

Surprising Application¶

Hiring and personnel decisions. Instead of one interviewer's judgment, use multiple independent evaluators with different perspectives (technical, cultural, managerial). Don't let them discuss before submitting scores - you need independent errors. The aggregated prediction outperforms any individual, even experts.

Failure Modes¶

Systematic bias - if all models make the same wrong assumption, ensembling doesn't help. Correlated errors - if models fail in the same scenarios, diversity is illusory. Computational cost - training and maintaining multiple models requires more resources. Interpretability loss - ensemble predictions are harder to explain than single models.

Go Deeper¶

Hastie et al., Elements of Statistical Learning, Chapter 15; Dietterich (2000), "Ensemble Methods in Machine Learning," MCS