Replication Hierarchy¶

Why It Matters¶

A single study, no matter how well-designed, has sampling error, unmeasured confounds, and researcher degrees of freedom. Replication reveals which findings are robust versus flukes or artifacts. Different replication types test different aspects: direct replication tests reliability, conceptual replication tests whether the effect survives different operationalizations, systematic replication maps the boundary conditions.

The Key Move¶

When evaluating evidence strength, count and categorize replications. One study = preliminary. Multiple direct replications = reliable under those specific conditions. Conceptual replications across methods/populations = generalizable phenomenon. Failed replications = null result, file-drawer problem, or false positive. Don't just ask "does it replicate?" - ask what type of replication and what it reveals.

Classic Application¶

Physics establishes findings through multiple independent labs using different equipment and methods. CERN's Higgs boson discovery required multiple detector teams analyzing different collision events with independent analysis pipelines. Agreement across fundamentally different measurement approaches is what establishes physical reality versus artifact.

Surprising Application¶

Personal decision rules and heuristics. You notice a pattern once ("I work better in the morning"). Direct replication: does it hold tomorrow? Next week? Conceptual replication: does it hold for different work types (coding vs. writing vs. meetings)? Systematic replication: what defines "morning" (8am? 10am? Before lunch?). Most people generalize from one instance; replication thinking demands systematic testing.

Failure Modes¶

Treating all replications as equivalent (direct replications are weakest evidence of generalization). Publication bias toward successful replications (file-drawer problem). Conducting "conceptual replications" so different they test different hypotheses. Using failed replication to claim the original was fraudulent rather than investigating moderators.

Go Deeper¶

Schmidt, S. (2009), "Shall we really do it again? The powerful concept of replication is neglected in the social sciences," Review of General Psychology 13(2); Open Science Collaboration (2015), "Estimating the reproducibility of psychological science," Science 349(6251).