Reliability and Validity Distinction¶

Why It Matters¶

People conflate precise with accurate. A scale that's calibrated wrong gives consistent readings (reliable) but wrong weight (invalid). A hiring rubric that everyone scores the same way is reliable, but if it doesn't predict job performance, it's invalid. This distinction clarifies that consistency in measurement is necessary but not sufficient - you must also verify the measurement captures the actual construct of interest.

The Key Move¶

For any measurement, ask two separate questions: (1) Reliability: "If I measure the same thing twice, do I get the same answer?" Test this with repeated measurements, multiple raters, or internal consistency checks. (2) Validity: "Does this measurement actually capture what I care about?" Test this by correlating with independent measures of the construct or checking if it predicts what it theoretically should.

Classic Application¶

Psychological test construction. An IQ test must be reliable (same person gets similar scores on retake) and valid (actually predicts performance on cognitive tasks, academic achievement, etc.). Inter-rater reliability ensures different scorers agree. Criterion validity ensures the test predicts real-world outcomes, not just internal test performance.

Surprising Application¶

Meeting effectiveness metrics. You track "meetings completed on time" (highly reliable - easy to measure consistently) as a proxy for "productive collaboration" (what you actually care about - validity unknown). The metric is reliable but possibly invalid: finishing on time says nothing about whether the meeting achieved useful outcomes. Better: directly measure what you care about, even if it's harder to measure reliably.

Failure Modes¶

Optimizing reliability at the expense of validity (measuring what's easy to measure consistently rather than what matters). Assuming face validity is sufficient (it "looks like" it measures the thing). Ignoring that validity is context-dependent (a valid measure in one population may be invalid in another). Using unreliable measures and wondering why results don't replicate.

Go Deeper¶

Cronbach & Meehl (1955), "Construct validity in psychological tests," Psychological Bulletin 52(4); Shadish, Cook, & Campbell (2002), Experimental and Quasi-Experimental Designs, Chapter 2.