Discriminating Test Selection¶

Why It Matters¶

Confirmation bias drives test selection toward confirming the favored hypothesis. This generates redundant information rather than discrimination. Discriminating test selection corrects the error of information-gathering strategies that reduce uncertainty slowly (or not at all) because they're optimized for confirmation rather than discrimination.

The Key Move¶

List your remaining hypotheses with their current probabilities. For each potential test, estimate its expected result under each hypothesis. Calculate the information gain - how much would this result change the probability distribution? Select the test with maximum expected information gain. The counterintuitive insight: the best test often has intermediate prior probability, not high or low, because it maximally separates hypotheses.

Classic Application¶

Fever workup with differential of pneumonia vs. pulmonary embolism vs. endocarditis. Chest X-ray is often ordered first because it's easy. But discriminating test selection: X-ray shows infiltrate in 70% of pneumonia, 30% of PE, 10% of endocarditis - moderate discrimination. D-dimer is negative in ~95% of non-PE cases and 0% of PE cases - extremely high discrimination for PE. Blood cultures discriminate perfectly for endocarditis but take days. Optimal sequence: d-dimer (fast, discriminates PE), then X-ray, then cultures if needed.

Surprising Application¶

Performance bottleneck identification. System is slow - could be CPU, memory, disk I/O, or network. Which metric to check first? CPU utilization discriminates poorly (high in most slowness scenarios). Disk I/O wait time discriminates strongly (high only if disk-bound, low otherwise). Check discriminating metric first, then narrow. The tool transfers: select observations that maximally separate hypotheses rather than measurements that are easy or confirm suspicions.

Failure Modes¶

Optimizing for pure information gain while ignoring test costs, risks, or delays. The best discriminating test might be invasive or expensive. The practical tool requires weighting information gain by cost and risk. Also fails when likelihood ratios for tests are unknown - you can't calculate discrimination without knowing test characteristics. The worst failure is paralysis from over-analysis - sometimes you must test sequentially rather than optimizing the sequence.

Go Deeper¶

Pauker, S.G. & Kassirer, J.P. (1975). "Therapeutic Decision Making: A Cost-Benefit Analysis," New England Journal of Medicine, 293(5), 229-234; Raiffa, H. & Schlaifer, R. (1961). "Applied Statistical Decision Theory," particularly on value of information.