Pre-Post Design with Control¶

Why It Matters¶

Simple pre-post comparisons confound treatment effects with natural change over time. Students improve from fall to spring - is it teaching or maturation? A company's revenue increases after a new strategy - is it the strategy or market growth? Without a control group experiencing the same time period, you can't tell. With it, you can isolate the treatment effect.

The Key Move¶

When evaluating any intervention over time, measure both treated and untreated groups before and after. Calculate the change in each group, then subtract: the treatment effect is the excess change relative to control. If both groups improve equally, the treatment did nothing. If treatment improves more, that difference is attributable to the intervention (assuming randomization or matched controls).

Classic Application¶

Educational program evaluation. To test if a new curriculum works, measure student achievement in fall and spring in both schools using the new curriculum (treatment) and similar schools using the old one (control). Both groups improve fall-to-spring (maturation), but if the treatment group improves 5% more, that's attributable to the curriculum.

Surprising Application¶

Evaluating personal habit changes. You start meditating daily and notice improved focus. But you also started exercising and sleeping better that month (confounds). Better: identify a control variable that should be unaffected by meditation (e.g., reaction time if meditation improves sustained attention but not processing speed). If meditation's effect is real, treated outcomes should change more than control outcomes.

Failure Modes¶

Assuming parallel trends - control and treatment groups must have similar trajectories absent treatment. Regression to the mean - selecting extreme cases guarantees apparent improvement. Differential dropout - if treatment group loses unmotivated members, the comparison becomes invalid. Ignoring that pre-testing can itself affect outcomes (testing effects).

Go Deeper¶

Imbens & Wooldridge (2009), "Recent developments in the econometrics of program evaluation," Journal of Economic Literature 47(1); Wing, Bello-Gomez, & Gourevitch (2018), Regression Discontinuity Designs, Chapter 4.