Subjective Probability Assessment in Decision Analysis: Partition Dependence and Bias Toward the Ignorance Prior

Decision and risk analysts have considerable discretion in designing procedures for eliciting subjective probabilities. One of the most popular approaches is to specify a particular set of exclusive and exhaustive events for which the assessor provides such judgments. We show that assessed probabilities are systematically biased toward a uniform distribution over all events into which the relevant state space happens to be partitioned, so that probabilities are "partition dependent." We surmise that a typical assessor begins with an "ignorance prior" distribution that assigns equal probabilities to all specified events, then adjusts those probabilities insufficiently to reflect his or her beliefs concerning how the likelihoods of the events differ. In five studies, we demonstrate partition dependence for both discrete events and continuous variables (Studies 1 and 2), show that the bias decreases with increased domain knowledge (Studies 3 and 4), and that top experts in decision analysis are susceptible to this bias (Study 5). We relate our work to previous research on the "pruning bias" in fault-tree assessment (e.g., Fischhoff et al. 1978) and show that previous explanations of pruning bias (enhanced availability of events that are explicitly specified, ambiguity in interpreting event categories, and demand effects) cannot fully account for partition dependence. We conclude by discussing implications for decision analysis practice.

[1]  G. Brier.  VERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITY , 1950 .

[2]  M. Orne.  On the social psychology of the psychological experiment: With particular reference to demand characteristics and their implications. , 1962 .

[3]  Daniel Kahneman,et al.  Availability: A heuristic for judging frequency and probability , 1973 .

[4]  A. Tversky,et al.  Judgment under Uncertainty: Heuristics and Biases , 1974, Science.

[5]  T. Fine,et al.  The Emergence of Probability , 1976 .

[6]  Carl-Axel S. Staël von Holstein,et al.  Exceptional Paper---Probability Encoding in Decision Analysis , 1975 .

[7]  Baruch Fischhoff,et al.  Calibration of Probabilities: The State of the Art , 1977 .

[8]  B. Fischhoff,et al.  Calibration of probabilities: the state of the art to 1980 , 1982 .

[9]  J. Eiser,et al.  COMPARATIVE JUDGMENTS AND PREFERENCES : THE INFLUENCE OF THE NUMBER OF RESPONSE ALTERNATIVES , 1987 .

[10]  M. W. Merkhofer,et al.  Quantifying judgmental uncertainty: Methodology, experiences, and insights , 1987, IEEE Transactions on Systems, Man, and Cybernetics.

[11]  E. Hirt,et al.  Probability and category redefinition in the fault tree paradigm. , 1988, Journal of experimental psychology. Human perception and performance.

[12]  Ronald A. Howard,et al.  Decision analysis: practice and promise , 1988 .

[13]  J. Edward Russo,et al.  An availability bias in professional judgment. , 1988 .

[14]  D. Winterfeldt,et al.  The effects of splitting attributes on weights in multiattribute utility measurement , 1988 .

[15]  Max Henrion,et al.  Uncertainty: A Guide to Dealing with Uncertainty in Quantitative Risk and Policy Analysis , 1990 .

[16]  O. Larichev,et al.  Contemporary issues in decision making , 1990 .

[17]  J. Pligt,et al.  Influence diagrams and fault trees: the role of salience and anchoring , 1990 .

[18]  H. Arkes.  Costs and benefits of judgment errors: Implications for debiasing. , 1991 .

[19]  F. Strack,et al.  Ease of retrieval as information: Another look at the availability heuristic. , 1991 .

[20]  G. Wells,et al.  Outcome trees and baseball: A study of expertise and list-length effects , 1991 .

[21]  Ralph L. Keeney,et al.  Eliciting probabilities from experts in complex technical problems , 1991 .

[22]  R. Cooke.  Experts in Uncertainty: Opinion and Subjective Probability in Science , 1991 .

[23]  Robert Sugden,et al.  Testing for juxtaposition and event-splitting effects , 1993 .

[24]  J. Edward Russo,et al.  Where is the fault in fault trees , 1994 .

[25]  K. Fiedler,et al.  Two halfs may be more than one whole : category-split effects on frequency illusions , 1994 .

[26]  A. Tversky,et al.  Support theory: A nonextensional representation of subjective probability. , 1994 .

[27]  Joop van der Pligt,et al.  Getting an anchor on availability in causal judgment , 1994 .

[28]  A. Tversky,et al.  Heuristics and Biases: Unpacking, Repacking, and Anchoring: Advances in Support Theory , 2002 .

[29]  Jack B. Soll,et al.  Overconfidence: It Depends on How, What, and Whom You Ask. , 1999, Organizational behavior and human decision processes.

[30]  Robert T. Clemen,et al.  Making Hard Decisions with Decisiontools Suite , 2000 .

[31]  A. Tversky,et al.  Choices, Values, and Frames , 2000 .

[32]  Ofir.  Ease of Recall vs Recalled Evidence in Judgment: Experts vs Laymen. , 2000, Organizational behavior and human decision processes.

[33]  Nigel Harvey,et al.  Are absolute frequencies, relative frequencies, or both effective in reducing cognitive biases? , 2000 .

[34]  R. Thaler,et al.  Naive Diversification Strategies in Defined Contribution Saving Plans , 2001 .

[35]  N. Epley,et al.  Putting Adjustment Back in the Anchoring and Adjustment Heuristic: Differential Processing of Self-Generated and Experimenter-Provided Anchors , 2001, Psychological science.

[36]  D. Kahneman,et al.  Heuristics and Biases: The Psychology of Intuitive Judgment , 2002 .

[37]  Eric J. Johnson,et al.  Incorporating the Irrelevant: Anchors in Judgments of Belief and Value , 2002 .

[38]  Craig R. Fox,et al.  Partition Priming in Judgment Under Uncertainty , 2003, Psychological science.

[39]  M. Birnbaum,et al.  Emerging Perspectives on Judgment and Decision Research: Generalization Across People, Procedures, and Predictions: Violations of Stochastic Dominance and Coalescing , 2003 .

[40]  M. Birnbaum.  Causes of Allais common consequence paradoxes: An experimental dissection ☆ , 2004 .

[41]  G. Wells,et al.  The Alternative-Outcomes Effect , 1998 .

[42]  Jonathan Levav,et al.  Partition-edit-count: naive extensional reasoning in judgment of conditional probability. , 2004, Journal of experimental psychology. General.

[43]  Yuval Rottenstreich,et al.  Typical versus atypical unpacking and superadditive probability judgment. , 2004, Journal of experimental psychology. Learning, memory, and cognition.

[44]  Nigel Harvey,et al.  Blackwell Handbook of Judgment and Decision Making , 2004 .

[45]  Daniel S. Lieb,et al.  Partition Dependence in Decision Analysis, Resource Allocation, and Consumer Choice , 2005 .

[46]  B. Fischhoff,et al.  Fault Trees: Sensitivity of Estimated Failure Probabilities to Problem Representation , 2005 .

[47]  Siobhan Chapman.  Logic and Conversation , 2005 .

[48]  Rebecca K. Ratner,et al.  How subjective grouping of options influences choice and allocation: diversification bias and the phenomenon of partition dependence. , 2005, Journal of experimental psychology. General.

[49]  A. Tversky,et al.  Rational choice and the framing of decisions , 1990 .