Skip to content

Class Imbalance and Base Rates

Why It Matters

Most real-world problems have imbalanced classes - fraud detection, disease diagnosis, equipment failure. Standard ML approaches optimize for overall accuracy, which is dominated by the majority class. This creates models that ignore rare but important events. Humans make the same error: neglecting base rates when evaluating evidence.

The Key Move

Never report accuracy alone for imbalanced problems - report precision, recall, and F1 for minority classes, or use metrics like AUROC that are insensitive to class balance. In training, weight minority class errors more heavily or balance the training data. When interpreting any classification, ask: "What was the base rate, and does this 'success rate' beat the trivial always-predict-majority baseline?"

Classic Application

Fraud detection. If 0.1% of transactions are fraudulent, a model that flags nothing as fraud achieves 99.9% accuracy but catches zero fraud. Proper approach: optimize for precision-recall tradeoff, where catching 50% of fraud with 10% false positive rate is vastly better than the naive baseline.

Surprising Application

Medical screening and testing. A test with 99% accuracy souscreening, but for a disease with 1% prevalence, most positive results are false positives. Understanding this base rate integration prevents both over-screening (chasing false positives) and under-appreciating negative results (which are highly informative for rare conditions).

Failure Modes

Artificially balanced training data can miscalibrate probability estimates - the model learns the wrong base rate. Cost-sensitive learning requires knowing actual cost ratios, which are often unclear. Changing base rates between training and deployment invalidate models.

Go Deeper

Provost (2000), "Machine Learning from Imbalanced Data Sets," AAAI Workshop; Fawcett (2006), "An Introduction to ROC Analysis," Pattern Recognition Letters