Testing Best Practices to Reduce the Overconfidence Bias in Multi-criteria Decision Analysis

This paper explores the effectiveness of several methods to reduce the overconfidence bias when eliciting continuous probability distributions in the context of multicriteria decision analysis. We examine the effectiveness of using a fixed value method (as opposed to the standard fixed probability method) and the use of counterfactuals and hypothetical bets to increase the range of the distributions and to correct possible median displacements. The results show that the betting procedure to correct the median is quite effective, but the methods to increase the range of estimates have only a have small, but positive effect.

[1]  Terrance Odean.  Do Investors Trade Too Much? , 1998 .

[2]  Ilan Yaniv,et al.  Overconfidence in interval estimates: What does expertise buy you? , 2008 .

[3]  B. Fischhoff,et al.  Training for calibration. , 1980 .

[4]  Pavlo R. Blavatskyy.  Betting on Own Knowledge: Experimental Test of Overconfidence , 2008 .

[5]  Richard H. Thaler,et al.  Chapter 13 Financial decision-making in markets and firms: A behavioral perspective , 1995, Finance.

[6]  B. Fischhoff,et al.  Assessing uncertainty in physical constants , 1986 .

[7]  Craig R. M. McKenzie,et al.  Underweighting Alternatives and Overconfidence , 1997 .

[8]  Kent D. Daniel,et al.  Presentation Slides for 'Investor Psychology and Security Market Under and Overreactions' , 1998 .

[9]  S. Oskamp.  OVERCONFIDENCE IN CASE-STUDY JUDGMENTS. , 1965, Journal of consulting psychology.

[10]  Vadim Subbotin,et al.  Outcome Feedback Effects on Under- and Overconfident Judgments (General Knowledge Tasks) , 1996 .

[11]  Thomas C. Omer,et al.  The appeal of vague financial forecasts , 2011 .

[12]  Magne Jørgensen,et al.  When 90% confidence intervals are 50% certain: on the credibility of credible intervals , 2005 .

[13]  Jay J.J. Christensen-Szalanski,et al.  Physicians' use of probabilistic information in a real clinical setting. , 1981 .

[14]  Joshua Klayman,et al.  Overconfidence in interval estimates. , 2004, Journal of experimental psychology. Learning, memory, and cognition.

[15]  Keith D. Markman,et al.  Multiple explanation: A consider-an-alternative strategy for debiasing judgments. , 1995 .

[16]  Hal R. Arkes,et al.  Overconfidence in Judgmental Forecasting , 2001 .

[17]  R. Thaler,et al.  Financial Decision-Making in Markets and Firms: A Behavioral Perspective , 1994 .

[18]  Anders Winman,et al.  The naïve intuitive statistician: a naïve sampling model of intuitive confidence intervals. , 2007, Psychological review.

[19]  D. Winterfeldt,et al.  Cognitive and Motivational Biases in Decision and Risk Analysis , 2015, Risk analysis : an official publication of the Society for Risk Analysis.

[20]  H. Arkes,et al.  Hindsight bias among physicians weighing the likelihood of diagnoses. , 1981, The Journal of applied psychology.

[21]  Ali E. Abbas,et al.  A Comparison of Two Probability Encoding Methods: Fixed Probability vs. Fixed Variable Values , 2008, Decis. Anal..

[22]  Dean P. Foster,et al.  Precision and Accuracy of Judgmental Estimation , 1997 .

[23]  A. Tversky,et al.  Judgment under Uncertainty: Heuristics and Biases , 1974, Science.

[24]  Carey K. Morewedge,et al.  A simple remedy for overprecision in judgment. , 2010, Judgment and decision making.

[25]  Robert T. Clemen,et al.  Interior Additivity and Subjective Probability Assessment of Continuous Variables , 2008, Manag. Sci..

[26]  R I Benjamin,et al.  Critical IT (information technology) issues: the next ten years. , 1992, Sloan management review.

[27]  Claudia González-Vallejo,et al.  Aligning confidence with accuracy: revisiting the role of feedback. , 2007, Acta psychologica.

[28]  A. Tversky,et al.  The Psychology of Preferences , 1982 .

[29]  B. Fischhoff,et al.  Judgment under uncertainty: Debiasing , 1982 .

[30]  R. L. Winkler,et al.  Scoring Rules for Continuous Probability Distributions , 1976 .

[31]  J. Baranski,et al.  The calibration and resolution of confidence in perceptual judgments , 1994, Perception & psychophysics.

[32]  A. Tversky,et al.  Prospect theory: analysis of decision under risk , 1979 .

[33]  Anders Winman,et al.  Subjective probability intervals: how to reduce overconfidence by interval evaluation. , 2004, Journal of experimental psychology. Learning, memory, and cognition.

[34]  Richard P. Larrick,et al.  Social comparison and confidence: When thinking you’re better than average predicts overconfidence (and when it does not) , 2007 .

[35]  B. Fischhoff,et al.  Reasons for confidence. , 1980 .

[36]  Gerd Gigerenzer,et al.  How to Improve Bayesian Reasoning Without Instruction: Frequency Formats , 1995 .

[37]  J V Baranski,et al.  Realism of confidence in sensory discrimination , 1999, Perception & psychophysics.

[38]  A. Tversky,et al.  Prospect theory: an analysis of decision under risk — Source link , 2007 .

[39]  D. Moore,et al.  The trouble with overconfidence. , 2008, Psychological review.

[40]  Don A. Moore,et al.  Overprecision in Judgment , 2015 .

[41]  Dean P. Foster,et al.  Graininess of judgment under uncertainty: An accuracy-informativeness trade-off , 1995 .

[42]  H. Raiffa,et al.  Judgment under uncertainty: A progress report on the training of probability assessors , 1982 .

[43]  Guo Ying Luo,et al.  An Experimental Test of the Impact of Overconfidence and Gender on Trading Activity , 2003 .

[44]  M. Carter,et al.  Obstetrician-gynecologists' decision making about the diagnosis of major depressive disorder and premenstrual dysphoric disorder , 2005, Journal of psychosomatic obstetrics and gynaecology.

[45]  T. Rakow,et al.  Improving calibration without training: the role of task information , 2003 .

[46]  David W. Keith,et al.  Improving the way we think about projecting future energy use and emissions of carbon dioxide , 2008 .

[47]  Jack B. Soll,et al.  Overconfidence: It Depends on How, What, and Whom You Ask. , 1999, Organizational behavior and human decision processes.

[48]  D. A. Seaver,et al.  Eliciting subjective probability distributions on continuous variables , 1978 .

[49]  Baruch Fischhoff,et al.  Calibration of Probabilities: The State of the Art , 1977 .