Skip to content

Structured Source Evaluation

Why It Matters

Source credibility massively affects evidence value, but humans are poor intuitive judges of credibility. We're swayed by confidence, specificity, and consistency with our beliefs rather than actual reliability indicators. Structured evaluation forces explicit consideration of access, motivation, track record, and corroboration.

The Key Move

For each source, evaluate:

  1. Access - are they in position to know this?
  2. Motivation - what incentives do they have to report accurately or deceive?
  3. Track record - have they provided reliable information before?
  4. Corroboration - is this consistent with other independent sources? Assign explicit credibility ratings (e.g., "usually reliable," "untested," "of doubtful reliability") and information reliability ratings ("confirmed," "probably true," "doubtful"). Report these separately from content - readers need to assess both what was said and who said it.

Classic Application

Intelligence reporting standards. NATO uses a standardized scale: source reliability (A-F, from "completely reliable" to "reliability cannot be judged") and information credibility (1-6, from "confirmed by other sources" to "truth cannot be judged"). This forces analysts to separate source assessment from content assessment and makes uncertainty explicit.

Surprising Application

Financial news consumption. Rather than taking headlines at face value, sophisticated investors evaluate: Does this outlet have access to company insiders or are they aggregating other sources? What incentives do they have (generating clicks vs. maintaining reputation)? What's their track record on this sector? Are multiple independent sources reporting this? This prevents acting on unreliable information just because it's widely reported.

Failure Modes

Conflating source confidence with information accuracy (confident sources can be wrong). Treating all information from a source as equally reliable (even good sources have variable access and occasional errors). Over-weighting recent performance (one good call doesn't make a source reliable). Under-weighting motivation to deceive (people believe sources they like). Not updating assessments based on track record (sources can become more or less reliable over time).

Go Deeper

"Intelligence Source and Information Reliability" (CIA, 2008). Lowenthal, "Intelligence: From Secrets to Policy" Chapter 5 on source evaluation. Marrin, "Improving Intelligence Analysis: Bridging the Gap Between Scholarship and Practice" Chapter 4.