Internal vs. External Validity Trade-off¶

Why It Matters¶

The perfect laboratory experiment achieves high internal validity but may test an artificial situation that never occurs naturally. Field studies have ecological validity but introduce confounds that reduce causal certainty. There's a fundamental trade-off: control versus realism. Recognizing this prevents fetishizing either extreme and makes you strategic about which to prioritize for different questions.

The Key Move¶

For any study, explicitly assess both validities. High internal validity: controlled conditions, random assignment, standardized measures. High external validity: natural settings, diverse samples, realistic interventions. Then ask: which matters more for this question? For mechanism discovery, prioritize internal. For policy recommendations, prioritize external. Ideally, pursue both through replication across contexts.

Classic Application¶

Drug development progression: Phase I trials prioritize internal validity (controlled hospital settings, careful monitoring, homogeneous samples). Phase III trials prioritize external validity (diverse populations, real-world treatment conditions, longer timescales). Both are necessary - mechanism understanding requires control, practical application requires generalization.

Surprising Application¶

A/B testing in product development. High internal validity: show variant B to random 50% of users, measure conversion. But external validity concerns: your current users aren't future users; this week's context isn't next year's; this feature change might interact with future changes. The clean A/B test answers "does this work now for these users" but not "should we ship this permanently for everyone."

Failure Modes¶

Assuming lab findings automatically generalize (psychologists' WEIRD sample problem). Dismissing controlled studies as "not realistic enough" - sometimes you need control to isolate mechanisms. Treating external validity as binary rather than a question of scope (generalizes to which contexts?). Ignoring that both validities are strengthened by replication, not by single perfect studies.

Go Deeper¶

Shadish, Cook, & Campbell (2002), Experimental and Quasi-Experimental Designs, Chapters 1-2; Henrich, Heine, & Norenzayan (2010), "The weirdest people in the world?" Behavioral and Brain Sciences 33(2-3).