Statistical Power Analysis¶
Why It Matters¶
Most people reason "we did the study, found no significant difference, therefore no effect exists." This is backwards when power is low. An underpowered study has a high false-negative rate - it will miss real effects. This generates a systematic bias toward null findings that has nothing to do with reality. Power analysis forces you to distinguish "we looked carefully and found nothing" from "we glanced briefly and didn't notice anything obvious."
The Key Move¶
Before running any experiment, calculate the minimum detectable effect given your sample size and measurement precision. Ask: "What's the smallest effect I could reliably detect?" Then compare this to the effect size you actually care about. If you can only detect huge effects but care about small ones, the study is uninformative. After a null finding, report power: "We had 80% power to detect a 10% improvement; we found a 2% improvement with p=0.4."
Classic Application¶
Clinical trials sample size determination. Before running the trial, calculate how many patients you need to detect a clinically meaningful improvement (e.g., 15% reduction in symptoms) with 80% power. If you can only recruit 50 patients but need 200 for adequate power, the trial shouldn't run - it will waste resources and likely produce an uninformative null result.
Surprising Application¶
Personal metrics and self-tracking. You weigh yourself daily and see fluctuations. Are you gaining weight or is it noise? Power analysis reveals that daily weight varies by ±2-3 pounds naturally, so detecting a real 5-pound change requires weeks of data. Looking at a single week is underpowered - you're mostly measuring water retention and measurement error, not actual trends.
Failure Modes¶
Calculating power based on hoped-for effect sizes rather than realistic ones (wishful thinking). Running the study anyway when power analysis shows it's underpowered (sunk cost fallacy). Using power analysis only after data collection to explain away null results (post-hoc rationalization). Confusing power with significance - they're different questions.
Go Deeper¶
Cohen, J. (1988), Statistical Power Analysis for the Behavioral Sciences, 2nd ed.; Button et al. (2013), "Power failure: why small sample size undermines the reliability of neuroscience," Nature Reviews Neuroscience 14(5).