Skip to content

Experimental Design

Foundational Control

Measurement and Detection

Confound Identification and Control

Replication and Generalization

Quick Reference

Decision Type → Tool Mapping

When you need to... Use these tools
Establish causation rather than correlation Controlled Comparison, Randomization, Confound Mapping
Determine if a finding is real or noise Statistical Power Analysis, Replication Hierarchy, Pre-Registration
Prevent bias in measurement Blinding, Reliability and Validity Distinction, Multiple Measurement
Evaluate whether an effect generalizes Internal vs. External Validity Trade-off, Boundary Condition Mapping
Test intervention effectiveness over time Pre-Post Design with Control, Dose-Response Testing
Make concepts testable Operational Definition
Verify findings independently Replication Hierarchy, Transparency
Identify alternative explanations Confound Mapping, Dose-Response Testing

Suggested Reading Path

  1. Entry point: Shadish, W.R., Cook, T.D., & Campbell, D.T. (2002). Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Houghton Mifflin. - Comprehensive yet accessible introduction to the logic of causal inference and experimental design.

  2. Deepening understanding: Pearl, J. (2009). Causality: Models, Reasoning, and Inference, 2nd edition. Cambridge University Press. - More technical treatment of causal reasoning using directed acyclic graphs and counterfactual frameworks.

  3. Modern context: Imbens, G.W. & Rubin, D.B. (2015). Causal Inference for Statistics, Social, and Biomedical Sciences. Cambridge University Press. - Potential outcomes framework with extensive treatment of observational studies and modern methods.

  4. Replication crisis and solutions: Open Science Collaboration (2015). "Estimating the reproducibility of psychological science." Science, 349(6251). Follow with the broader literature on pre-registration, replication, and open science practices.

  5. Historical foundations: Fisher, R.A. (1935). The Design of Experiments. Oliver & Boyd. - Original source for randomization, null hypothesis testing, and experimental logic. Dated but intellectually foundational.

Usage Notes

Domain of applicability: These tools work best when you have control over assignment (can randomize), can manipulate the independent variable, and can measure outcomes relatively soon after intervention. They excel in medicine, psychology, agriculture, product development, and anywhere you can run controlled comparisons. They struggle with historical questions, rare events, ethical constraints on manipulation, or outcomes with very long time horizons.

Limitations: Experimental design tools don't tell you what hypotheses to test or what variables matter - they assume you've already identified relevant constructs. They require resources (time, money, samples) that may not be available. Perfect randomization and control are often impossible in field settings. These tools isolate causation but may sacrifice ecological validity. They work better for simple causal chains than complex systems with feedback loops and emergence.

Composition: Controlled comparison, randomization, and blinding form the core triad - use all three whenever possible for strongest inference. Statistical power analysis should precede every study. Confound mapping informs which variables to measure and control. Multiple measurement strengthens any design. Pre-registration prevents self-deception in all the above. The tools are complementary, not substitutes - each addresses a different threat to valid inference.

Integration: Experimental design tools integrate naturally with Bayesian reasoning (prior beliefs + experimental evidence → posterior beliefs), systems thinking (experiments test hypotheses about causal structure), and decision analysis (experiments reduce uncertainty for better decisions). They complement but differ from observational methods (epidemiology, natural experiments) which face different trade-offs. They provide the epistemological foundation for evidence-based medicine, program evaluation, and data-driven product development.

Practical advice: Start with the clearest possible comparison (Tier 1), ensure you can detect effects if they exist (power analysis), eliminate or measure confounds, then test robustness through replication and boundary mapping. When you can't run true experiments, use quasi-experimental designs and explicitly acknowledge limitations. Always ask "compared to what?" and make the counterfactual explicit. Document decisions before seeing results to prevent self-deception.