Controlled Comparison¶

Why It Matters¶

Human intuition systematically overweights salient causes and underweights base rates. We see someone take a supplement and get better, and conclude the supplement works - ignoring that most people get better anyway. Controlled comparison forces you to ask "compared to what?" and makes the counterfactual explicit. Without it, you're engaging in storytelling, not causal inference.

The Key Move¶

When evaluating any causal claim "X causes Y," identify the comparison condition: what happens when X is absent but everything else is the same? If you can't articulate this comparison, you don't have evidence of causation. Make the comparison explicit and concrete: not "the treatment worked" but "the treatment group improved 15% more than the control group."

Classic Application¶

Clinical drug trials. To test if a drug works, give it to one group and a placebo to another matched group. The difference in outcomes is attributable to the drug. Without the placebo control, you can't distinguish drug effects from natural recovery, placebo effects, or regression to the mean.

Surprising Application¶

Evaluating your own decision-making processes. You can't tell if your decision method works by only examining decisions you made - you need to compare outcomes to what would have happened with a different method. This explains why personal experience is weak evidence: you don't have access to the counterfactual "you" who made different choices. Career advice is notoriously unreliable because successful people compare their path to nothing, not to alternatives.

Failure Modes¶

Failing to hold "everything else" constant - uncontrolled comparisons are worthless. Comparing to unrealistic counterfactuals (e.g., "compared to doing nothing" when nobody would do nothing). Ignoring that the comparison condition might itself have effects (active controls matter). Treating the absence of a formal control as evidence rather than as absence of evidence.

Go Deeper¶

Fisher, R.A. (1935), The Design of Experiments, Chapter 2; Shadish, Cook, & Campbell (2002), Experimental and Quasi-Experimental Designs for Generalized Causal Inference, Chapter 1.