Implications of the research on expert overconfidence and dependence

Abstract In cases where interval estimates are obtained from expert opinion, the literature on expert overconfidence and dependence suggests that caution is in order. In particular, it may be inadvisable in such cases to interpret any areas of overlap among the intervals provided by the various experts as strong evidence that the quantity of interest falls within the region of overlap. The available literature on methods for assessing and compensating for overconfidence and for combining intervals or probability distributions from multiple experts is discussed. An ad hoc method of combining multiple expert opinions, suitable for use in relatively small projects, is also presented.

[1]  Baruch Fischhoff,et al.  How Well do Probability Experts Assess Probabilities , 1980 .

[2]  Robert L. Winkler,et al.  The Consensus of Subjective Probability Distributions , 1968 .

[3]  Thomas B. Sheridan,et al.  Filtering information from human experts , 1989, IEEE Trans. Syst. Man Cybern..

[4]  A. Tversky,et al.  Judgment under Uncertainty: Heuristics and Biases , 1974, Science.

[5]  B. Fischhoff,et al.  Reasons for confidence. , 1980 .

[6]  J. Hammitt,et al.  The Expected Value of Information and the Probability of Surprise , 1999 .

[7]  Ali Mosleh,et al.  Methods for the elicitation and use of expert opinion in risk assessment: Phase 1, A critical evaluation and directions for future research , 1987 .

[8]  M. M. Gupta,et al.  Uncertainty Estimates in Scientific Models : Lessons from Trends in Physical Measurements , Population and Energy Projections , 2002 .

[9]  Robert L. Winkler,et al.  Probabilistic Prediction: Some Experimental Results , 1971 .

[10]  G. W. Fischer.  When oracles fail—A comparison of four procedures for aggregating subjective probability forecasts , 1981 .

[11]  B. Fischhoff,et al.  Training for calibration. , 1980 .

[12]  David H. Gustafson,et al.  Comparison of rules for aggregating subjective likelihood ratios , 1974 .

[13]  P. Juslin,et al.  Format dependence in subjective probability calibration , 1999 .

[14]  Jon C. Helton,et al.  Challenge Problems : Uncertainty in System Response Given Uncertain Parameters ( DRAFT : November 29 , 2001 ) , 2001 .

[15]  Robert L. Winkler,et al.  Combining Probability Distributions From Experts in Risk Analysis , 1999 .

[16]  R. L. Winkler.  Combining Probability Distributions from Dependent Information Sources , 1981 .

[17]  R. Clemen,et al.  Correlations and Copulas for Decision and Risk Analysis , 1999 .

[18]  A. I. Shlyakhter,et al.  Estimating uncertainty in physical measurements, observational and environmental studies: lessons from trends in nuclear data , 1993, 1993 (2nd) International Symposium on Uncertainty Modeling and Analysis.

[19]  Henrik Olsson,et al.  Naive empiricism and dogmatism in confidence research: a critical examination of the hard-easy effect. , 2000 .

[20]  Robin M. Hogarth,et al.  A note on aggregating opinions , 1978 .

[21]  George Apostolakis,et al.  Some approximations useful to the use of dependent information sources , 1993 .

[22]  Robert T. Clemen,et al.  Copula Models for Aggregating Expert Opinions , 1996, Oper. Res..

[23]  S. Plous.  The psychology of judgment and decision making , 1994 .

[24]  S. Lichtenstein,et al.  Do those who know more also know more about how much they know?*1 , 1977 .

[25]  R. B. Kearfott,et al.  Applications of interval computations , 1996 .

[26]  William R. Ferrell,et al.  Combining Individual Judgments , 1985 .

[27]  G. William Walster,et al.  A comparative study of differences in subjective likelihood estimates made by individuals, interacting groups, Delphi groups, and nominal groups☆ , 1973 .

[28]  Carl-Axel S. Staël von Holstein,et al.  Probabilistic forecasting: An experiment related to the stock market , 1972 .

[29]  P. Juslin,et al.  Naive empiricism and dogmatism in confidence research: a critical examination of the hard-easy effect. , 2000, Psychological review.

[30]  R. Cooke.  Experts in Uncertainty: Opinion and Subjective Probability in Science , 1991 .

[31]  J. Charles Kerkering,et al.  Eliciting and Analyzing Expert Judgment, A Practical Guide , 2002, Technometrics.

[32]  G. Gigerenzer,et al.  Probabilistic mental models: a Brunswikian theory of confidence. , 1991, Psychological review.

[33]  W. Edwards,et al.  Decision Analysis and Behavioral Research , 1986 .

[34]  Jack B. Soll,et al.  Overconfidence: It Depends on How, What, and Whom You Ask. , 1999, Organizational behavior and human decision processes.

[35]  C. Holstein.  An Experiment in Probabilistic Weather Forecasting , 1971 .

[36]  W. E. Riggs,et al.  The Delphi technique: An experimental evaluation , 1983 .

[37]  I. Erev,et al.  Simultaneous Over- and Underconfidence: The Role of Error in Judgment Processes. , 1994 .

[38]  Christian Genest,et al.  Combining Probability Distributions: A Critique and an Annotated Bibliography , 1986 .

[39]  H. Raiffa,et al.  Judgment under uncertainty: A progress report on the training of probability assessors , 1982 .

[40]  Robert J. Genetski,et al.  Long-Range Forecasting: From Crystal Ball to Computer , 1981 .

[41]  Harry F. Martz.  On Broadening Failure Rate Distributions in PRA Uncertainty Analyses , 1984 .

[42]  T. A. Brown,et al.  An Experiment in Probabilistic Forecasting , 1973 .

[43]  B. Fischhoff,et al.  Assessing uncertainty in physical constants , 1986 .

[44]  Joseph P. Martino,et al.  LONG RANGE FORECASTING METHODOLOGY , 1968 .

[45]  A. I. Shlyakhter,et al.  An improved framework for uncertainty analysis: Accounting for unsuspected errors , 1994 .

[46]  P. Walley.  Statistical Reasoning with Imprecise Probabilities , 1990 .

[47]  R. H. Edmundson,et al.  The accuracy of combining judgemental and statistical forecasts , 1986 .

[48]  Daniel M. Kammen,et al.  Sea-level rise or fall? , 1992, Nature.

[49]  B. Fischhoff,et al.  Calibration of probabilities: the state of the art to 1980 , 1982 .

[50]  H. A. Lingstone,et al.  The Delphi Method: Techniques and Applications , 1976 .

[51]  Robert L. Winkler,et al.  Limits for the Precision and Value of Information from Dependent Sources , 1985, Oper. Res..

[52]  Robert L. Winkler,et al.  Assessing Dependence: Some Experimental Results , 2000 .