Diagnostic vs. Supportive Evidence¶

Why It Matters¶

Analysts naturally accumulate evidence that supports their hypothesis without asking whether that evidence would also support competing explanations. Supportive evidence creates false confidence. Diagnostic evidence - evidence that would be unlikely under alternative hypotheses - is what actually narrows uncertainty.

The Key Move¶

For each piece of evidence, ask: "If my hypothesis is wrong, would I still expect to see this evidence?" If yes, it's supportive but not diagnostic - it doesn't help you choose between hypotheses. If no, it's diagnostic - it genuinely distinguishes your hypothesis from alternatives. Focus analytical effort on finding diagnostic evidence and weigh it more heavily in your judgment.

Classic Application¶

Assessing whether a foreign official is a recruited intelligence asset or a double agent. Many behaviors (providing information, accepting money, covert communication) are consistent with both. Diagnostic evidence: information the source provides that their sponsoring intelligence service wouldn't want revealed, or that's contrary to their service's strategic interests. This distinguishes genuine recruitment from deception.

Surprising Application¶

Hiring decisions. A candidate's enthusiasm for the role is supportive evidence of fit but not diagnostic - poor fits are also enthusiastic during interviews. Diagnostic evidence: specific examples of behavior in situations similar to role challenges (they've solved analogous problems before) or evidence that would be unlikely if they're a poor fit (references who describe specific relevant behaviors without prompting).

Failure Modes¶

Treating all evidence as equally valuable (most evidence is supportive, not diagnostic). Ignoring that absence of evidence can be diagnostic (if you'd expect to see something under hypothesis A but not B, its absence favors B). Confusing "consistent with" (supportive) with "evidence for" (diagnostic). Not specifying in advance what evidence would be diagnostic (post-hoc rationalization). Assuming more supportive evidence compensates for lack of diagnostic evidence (quantity doesn't substitute for quality).

Go Deeper¶

Heuer, "Psychology of Intelligence Analysis" Chapter 4. Schum, "The Evidential Foundations of Probabilistic Reasoning" Chapter 4 on evidence weight. Pearl, "Causality" on evidential reasoning and d-separation.