Known Unknowns: A Critical Determinant of Confidence and Calibration

We propose that an important determinant of judged confidence is the evaluation of evidence that is unknown or missing, and overconfidence is often driven by the neglect of unknowns. We contrast this account with prior research suggesting that overconfidence is due to biased processing of known evidence in favor of a focal hypothesis. In Study 1, we asked participants to list their thoughts as they answered two-alternative forced-choice trivia questions and judged the probability that their answers were correct. Participants who thought more about unknowns were less overconfident. In Studies 2 and 3, we asked participants to list unknowns before assessing their confidence. “Considering the unknowns” reduced overconfidence substantially and was more effective than the classic “consider the alternative” debiasing technique. Moreover, considering the unknowns selectively reduced confidence in domains where participants were overconfident but did not affect confidence in domains where participants were well-c...

[1]  Derek J. Koehler,et al.  Modeling patterns of probability calibration with random support theory: Diagnosing case-based judgment ☆ , 2005 .

[2]  Derek J. Koehler,et al.  Heuristics and Biases: The Calibration of Expert Judgment: Heuristics and Biases Beyond the Laboratory , 2002 .

[3]  A. Tversky,et al.  Support theory: A nonextensional representation of subjective probability. , 1994 .

[4]  B. Fischhoff,et al.  Reasons for confidence. , 1980 .

[5]  Eric J. Johnson,et al.  Aspects of Endowment: A Query Theory of Value Construction , 2007, Journal of experimental psychology. Learning, memory, and cognition.

[6]  A. Tversky,et al.  Judgment under Uncertainty: Heuristics and Biases , 1974, Science.

[7]  Jayashree Mahajan.  The Overconfidence Effect in Marketing Management Predictions , 1992 .

[8]  Kelly E. See,et al.  Between ignorance and truth: Partition dependence and learning in judgment under uncertainty. , 2006, Journal of experimental psychology. Learning, memory, and cognition.

[9]  C. Fox.  Strength of Evidence, Judged Probability, and Choice Under Uncertainty , 1999, Cognitive Psychology.

[10]  GUlden Ulkiimen,et al.  DISTINGUISHING TWO DIMENSIONS OF UNCERTAINTY , 2011 .

[11]  Fergus Bolger,et al.  The calibration of subjective probability: Theories and models 1980–94. , 1994 .

[12]  Jack B. Soll,et al.  Overconfidence: It Depends on How, What, and Whom You Ask. , 1999, Organizational behavior and human decision processes.

[13]  Robert T. Clemen,et al.  Subjective Probability Assessment in Decision Analysis: Partition Dependence and Bias Toward the Ignorance Prior , 2005, Manag. Sci..

[14]  G. Gigerenzer,et al.  Probabilistic mental models: a Brunswikian theory of confidence. , 1991, Psychological review.

[15]  Craig R. Fox,et al.  Ambiguity Aversion, Comparative Ignorance, and Decision Context , 2002 .

[16]  Craig R. Fox,et al.  Ambiguity Aversion and Comparative Ignorance , 1995 .

[17]  Colin Camerer,et al.  Overconfidence and Excess Entry: An Experimental Approach , 1999 .

[18]  Ulrike Malmendier,et al.  CEO Overconfidence and Corporate Investment , 2002 .

[19]  Joshua Klayman,et al.  Overconfidence in interval estimates. , 2004, Journal of experimental psychology. Learning, memory, and cognition.

[20]  Lyle Brenner,et al.  Perspectives on Probability Judgment Calibration , 2008 .

[21]  Clifford R. Mynatt,et al.  Confirmation Bias in a Simulated Research Environment: An Experimental Study of Scientific Inference , 1977 .

[22]  Frank R. Kardes,et al.  The role of prior knowledge and missing information in multiattribute evaluation , 1992 .

[23]  M. R. Gordon,et al.  Cobra II: The Inside Story of the Invasion and Occupation of Iraq , 2006 .

[24]  Charles R. Schwenk,et al.  Effects of the expert, devil's advocate, and dialectical inquiry methods on prediction performance , 1980 .

[25]  J. Klayman.  Varieties of Confirmation Bias , 1995 .

[26]  I. Erev,et al.  Simultaneous Over- and Underconfidence: The Role of Error in Judgment Processes. , 1994 .

[27]  Elisha Y. Babad,et al.  Wishful thinking and objectivity among sports fans. , 1987 .

[28]  B. Fischhoff,et al.  Fault Trees: Sensitivity of Estimated Failure Probabilities to Problem Representation , 2005 .

[29]  R. Nickerson.  Confirmation Bias: A Ubiquitous Phenomenon in Many Guises , 1998 .

[30]  Richard P. Larrick,et al.  Motivational factors in decision theories : the role of self-protection , 1993 .

[31]  H. Raiffa,et al.  Judgment under uncertainty: A progress report on the training of probability assessors , 1982 .

[32]  William R. Ferrell,et al.  A model of calibration for subjective probabilities , 1980 .

[33]  Richard P. Larrick,et al.  Biased Judgment in Censored Environments , 2013, Manag. Sci..

[34]  A. Tversky,et al.  BELIEF IN THE LAW OF SMALL NUMBERS , 1971, Pediatrics.

[35]  D. Moore,et al.  The trouble with overconfidence. , 2008, Psychological review.

[36]  R. Deber,et al.  Overconfidence among physicians and nurses: the 'micro-certainty, macro-uncertainty' phenomenon. , 1991, Social science & medicine.

[37]  A. Tversky,et al.  The weighing of evidence and the determinants of confidence , 1992, Cognitive Psychology.

[38]  A. Tversky,et al.  On the psychology of prediction , 1973 .

[39]  Leonid Rozenblit,et al.  The misunderstood limits of folk science: an illusion of explanatory depth , 2002, Cogn. Sci..

[40]  Stephen J. Hoch,et al.  Counterfactual reasoning and accuracy in predicting personal events. , 1985 .

[41]  Craig R. Fox,et al.  Judgment Extremity and Accuracy Under Epistemic vs. Aleatory Uncertainty , 2017, Manag. Sci..

[42]  A. Tversky,et al.  Preference and belief: Ambiguity and competence in choice under uncertainty , 1991 .

[43]  Jeffrey C. Zemla,et al.  Missing the trees for the forest: a construal level account of the illusion of explanatory depth. , 2010, Journal of personality and social psychology.

[44]  Z. Kunda,et al.  The case for motivated reasoning. , 1990, Psychological bulletin.

[45]  A. Tversky,et al.  Intuitive Prediction: Biases and Corrective Procedures , 1982 .

[46]  R. Buehler,et al.  Exploring the "planning fallacy": Why people underestimate their task completion times. , 1994 .

[47]  D. Kahneman,et al.  Heuristics and Biases: The Psychology of Intuitive Judgment , 2002 .

[48]  A. Tversky,et al.  Heuristics and Biases: Unpacking, Repacking, and Anchoring: Advances in Support Theory , 2002 .

[49]  J. Klayman,et al.  Confirmation, Disconfirmation, and Informa-tion in Hypothesis Testing , 1987 .

[50]  B. Newell.  Judgment Under Uncertainty , 2013 .

[51]  Philip M. Fernbach,et al.  Asymmetries in predictive and diagnostic reasoning. , 2011, Journal of experimental psychology. General.

[52]  Itzhak Ben-David,et al.  Managerial Overconfidence and Corporate Policies , 2007 .

[53]  D. Kahneman.  Thinking, Fast and Slow , 2011 .

[54]  R. Scheines,et al.  Organizational Behavior and Human Decision Processes , 1977 .

[55]  S. Oskamp.  OVERCONFIDENCE IN CASE-STUDY JUDGMENTS. , 1965, Journal of consulting psychology.

[56]  Asher Koriat,et al.  The self-consistency model of subjective confidence. , 2012, Psychological review.

[57]  Steven A. Sloman,et al.  Political Extremism Is Supported by an Illusion of Understanding , 2013, Psychological science.

[58]  Jack B. Soll.  Determinants of Overconfidence and Miscalibration: The Roles of Random Error and Ecological Structure☆ , 1996 .

[59]  Brian D. Laslie,et al.  Cobra II: The Inside Story of the Invasion and Occupation of Iraq , 2007 .

[60]  C. Sansone,et al.  Remembering less and inferring more: effects of time of judgment on inferences about unknown attributes. , 1991, Journal of personality and social psychology.

[61]  Johnson,et al.  Overconfidence and War , 2004 .

[62]  John G. Lynch,et al.  Spotlights, Floodlights, and the Magic Number Zero: Simple Effects Tests in Moderated Regression , 2012 .