Skip to content

Expert Consultation Strategy

Why It Matters

Experts have extensive tacit knowledge they can't directly articulate. Direct questions ("how do you do this?") produce misleading answers - post-hoc rationalizations, over-simplified procedures, or advice that works for experts but not learners. Skilled elicitation reveals the actual mental operations, not just the surface behaviors or stated beliefs.

The Key Move

When learning from experts, avoid "how do you..." questions. Instead:

  1. Observe performance and ask them to verbalize thinking in real-time (think-aloud protocol).
  2. Present pairs of similar cases with different outcomes and ask what distinguishes them (contrastive cases).
  3. Ask about edge cases and exceptions ("when does this rule not apply?").
  4. Request predictions with explanations before revealing outcomes.
  5. Analyze errors together - experts often reveal their models through error diagnosis.

Classic Application

Knowledge engineering for expert systems. Early AI efforts failed by asking experts to state their decision rules. Effective approach: observe experts working on real cases, interrupt for explanation at decision points, present carefully designed test cases that distinguish competing rules, and build decision trees from contrastive analysis rather than direct rule elicitation. This revealed that experts used complex pattern-matching, not the simple rules they claimed.

Surprising Application

Professional mentorship. Junior professionals often ask mentors "how should I handle X?" and get generic advice. More effective: bring specific cases, present your analysis and decision, ask the mentor to explain where their thinking diverges and why (contrastive analysis). Or shadow the mentor during real decisions and ask for thought process explanations in the moment. This reveals the actual decision criteria and situational factors that generic advice obscures.

Failure Modes

Over-reliance on one expert's model when multiple valid approaches exist. Confusing what experts say with what they do (verbalized versus actual process). Insufficient expertise in the person being consulted - mid-level performers may have better explicit knowledge than true experts. Cultural or organizational barriers to revealing actual decision processes.

Go Deeper

Ericsson, K.A., & Simon, H.A. (1993). Protocol Analysis: Verbal Reports as Data (revised edition); Hoffman, R.R., Shadbolt, N.R., Burton, A.M., & Klein, G. (1995). "Eliciting knowledge from experts: A methodological analysis." Organizational Behavior and Human Decision Processes 62(2), 129-158.