The Effects of Presenting Imprecise Probabilities in Intelligence Forecasts

How to assess and present analytic uncertainty to policymakers has emerged as an important topic in risk and policy analysis. Due to the complexity and deep uncertainty present in many forecasting domains, these reports are often fraught with analytic uncertainty. In three studies, we explore the effect of presenting probability assessments and analytic uncertainty through probability ranges. Participants were presented with mock intelligence forecasts that include narrative evidence as well as numerical probability assessments. Participants were sensitive to the ambiguity communicated through the confidence range. The narrative appeared to have a smaller effect on judgments when accompanied by a probability range as opposed to a point assessment. In one study, participants also thought that the probability range was more useful for decision making at a higher probability whereas the point estimate was more useful at a lower probability. When evaluating a forecast in hindsight, decisionmakers tended to report lower levels of blame and higher levels of source credibility for forecasts that reported ranges as compared to point assessments. These findings suggest that decisionmakers are not necessarily "ambiguity averse" in the forecasting context. Presenting ranges of probability may have distinct advantages as a way to communicate probability and analytic confidence to decisionmakers.

[1]  M. Bar-Hillel,et al.  How alike is it versus how likely is it: A disjunction fallacy in probability judgments. , 1993 .

[2]  Colin Camerer,et al.  Recent developments in modeling preferences: Uncertainty and ambiguity , 1992 .

[3]  B. Fischhoff,et al.  Hindsight is not equal to foresight: The effect of outcome knowledge on judgment under uncertainty. , 1975 .

[4]  Paul Slovic,et al.  Lay views on uncertainty in environmental health risk assessment , 1998 .

[5]  D. Ellsberg.  Decision, probability, and utility: Risk, ambiguity, and the Savage axioms , 1961 .

[6]  R. Hogarth,et al.  Decision Making under Ambiguity , 1986 .

[7]  Adrian K. Rantilla,et al.  Confidence in aggregation of expert opinions. , 2000, Acta psychologica.

[8]  Kimberly M Thompson,et al.  Variability and Uncertainty Meet Risk Management and Risk Communication , 2002, Risk analysis : an official publication of the Society for Risk Analysis.

[9]  P. Slovic,et al.  The Use of Narrative Evidence and Explicit Likelihood by Decisionmakers Varying in Numeracy , 2009, Risk analysis : an official publication of the Society for Risk Analysis.

[10]  M. Elisabeth Paté-Cornell,et al.  Uncertainties in risk analysis: Six levels of treatment , 1996 .

[11]  Dean P. Foster,et al.  Graininess of judgment under uncertainty: An accuracy-informativeness trade-off , 1995 .

[12]  L. Robin Keller,et al.  Managers' variance investigation decisions: an experimental examination of probabilistic and outcome ambiguity , 2001 .

[13]  Shawn P. Curley,et al.  An Investigation of Patient's Reactions to Therapeutic Uncertainty , 1984 .

[14]  Max Henrion,et al.  Uncertainty: A Guide to Dealing with Uncertainty in Quantitative Risk and Policy Analysis , 1990 .

[15]  R. Hogarth,et al.  Venture Theory: A Model of Decision Weights , 1990 .

[16]  P. Slovic,et al.  Presenting uncertainty in health risk assessment: initial studies of its effects on risk perception and trust. , 1995, Risk analysis : an official publication of the Society for Risk Analysis.

[17]  Stephen J. Hoch,et al.  Consumer Learning: Advertising and the Ambiguity of Product Experience , 1986 .

[18]  S. Bonaccio,et al.  Advice taking and decision-making: An integrative literature review, and implications for the organizational sciences , 2006 .

[19]  Nils-Eric Sahlin,et al.  Decision making with unreliable probabilities , 1983 .

[20]  Joanna L. Y. Ho,et al.  Effects of Outcome and Probabilistic Ambiguity on Managerial Choices , 2002 .

[21]  Lyn M Van Swol,et al.  Factors affecting the acceptance of expert advice. , 2005, The British journal of social psychology.

[22]  David V. Budescu,et al.  Decisions based on numerically and verbally expressed uncertainties. , 1988 .

[23]  Dean P. Foster,et al.  Precision and Accuracy of Judgmental Estimation , 1997 .

[24]  R. Hastie,et al.  Hindsight: Biased judgments of past events after the outcomes are known. , 1990 .

[25]  Christopher K. Hsee.  Elastic Justification: How Tempting But Task-Irrelevant Factors Influence Decisions , 1995 .

[26]  R. Hogarth,et al.  Ambiguity and Uncertainty in Probabilistic Inference. , 1985 .

[27]  Craig W. Trumbo,et al.  Source Credibility in Environmental Health – Risk Controversies: Application of Meyer's Credibility Index , 2001, Risk analysis : an official publication of the Society for Risk Analysis.

[28]  Shawn P. Curley,et al.  The center and range of the probability interval as factors affecting ambiguity preferences , 1985 .

[29]  V. Reyna,et al.  How numeracy influences risk comprehension and medical decision making. , 2009, Psychological bulletin.

[30]  Charles F. Doran,et al.  Why Forecasts Fail: The Limits and Potential of Forecasting in International Relations and Economics , 1999 .

[31]  Craig R. Fox,et al.  The Role of Context in the Communication of Uncertain Beliefs , 1998 .

[32]  Rakesh K. Sarin,et al.  Modeling Ambiguity in Decisions Under Uncertainty , 1988 .

[33]  B. Johnson.  Further Notes on Public Response to Uncertainty in Risks and Science , 2003, Risk analysis : an official publication of the Society for Risk Analysis.