Skip to content

Prior Sensitivity Analysis

Why It Matters

Critics of Bayesian reasoning object that priors are subjective and therefore conclusions are arbitrary. Prior sensitivity analysis tests this: if your conclusion is robust across all reasonable priors, then prior choice doesn't matter - you have "prior-proof" evidence. If conclusions are sensitive, this reveals that evidence is weak relative to prior uncertainty - an important finding in itself. The tool converts "subjectivity" from a bug into a feature: it quantifies how much your conclusions depend on assumptions.

The Key Move

After reaching a Bayesian conclusion, ask: "What if my prior had been different?" Choose a range of plausible priors (optimistic, pessimistic, neutral) and recompute posteriors. If all reasonable priors yield similar posteriors, your conclusion is evidence-driven and robust. If priors matter a lot, either: (1) get more/better evidence, or (2) acknowledge that reasonable people with different priors will reach different conclusions from the same evidence. Make the sensitivity explicit in your reasoning.

Classic Application

Clinical trials for drug effectiveness. Prior beliefs about drug efficacy might range from "probably harmful" (pessimistic) to "probably beneficial" (optimistic) depending on previous studies. If the trial data is strong, all reasonable priors converge to similar posteriors (e.g., 70-80% probability of benefit). If data is weak, priors matter enormously (pessimistic prior → 30% probability of benefit; optimistic prior → 70%). Sensitivity analysis reveals whether the trial resolved uncertainty or whether interpretation depends on prior beliefs.

Surprising Application

Evaluating life decisions under uncertainty. Should you switch careers? Your prior on "career switch will improve happiness" depends on personality, risk tolerance, current satisfaction. Run sensitivity analysis: assume optimistic prior (70% success), pessimistic (30% success), neutral (50%). Consider evidence: informational interviews, skill assessment, financial modeling. If all priors yield similar conclusions given evidence, the decision is clear. If conclusions flip based on priors, you have insufficient evidence - and the decision depends on your risk preferences, not "objective analysis."

Failure Modes

Artificially narrow prior ranges - if you only test priors you already agree with, sensitivity analysis provides false confidence. Ignoring implausible priors that matter - sometimes a 1% prior probability on a catastrophic outcome dominates expected value; rare priors can matter. Over-interpreting robustness - stable posteriors might reflect weak evidence (all priors stay near 50%) not strong evidence. Forgetting model sensitivity - varying priors while keeping the model fixed ignores model uncertainty.

Go Deeper

Gelman et al., Bayesian Data Analysis, Chapter 6 on model checking and sensitivity; Berger, Statistical Decision Theory and Bayesian Analysis, Chapter 4.7.