The effects of averaging subjective probability estimates between and within judges.

The average probability estimate of J > 1 judges is generally better than its components. Two studies test 3 predictions regarding averaging that follow from theorems based on a cognitive model of the judges and idealizations of the judgment situation. Prediction 1 is that the average of conditionally pairwise independent estimates will be highly diagnostic, and Prediction 2 is that the average of dependent estimates (differing only by independent error terms) may be well calibrated. Prediction 3 contrasts between- and within-subject averaging. Results demonstrate the predictions' robustness by showing the extent to which they hold as the information conditions depart from the ideal and as J increases. Practical consequences are that (a) substantial improvement can be obtained with as few as 2-6 judges and (b) the decision maker can estimate the nature of the expected improvement by considering the information conditions.

[1]  Max Henrion,et al.  Uncertainty: A Guide to Dealing with Uncertainty in Quantitative Risk and Policy Analysis , 1990 .

[2]  H. Olsson,et al.  Underconfidence in sensory discrimination: The interaction between experimental setting and response strategies , 1996, Perception & psychophysics.

[3]  Robin M. Hogarth,et al.  On combining diagnostic ‘forecasts’: Thoughts and some evidence , 1989 .

[4]  P. Juslin.  The Overconfidence Phenomenon as a Consequence of Informal Experimenter-Guided Selection of Almanac Items , 1994 .

[5]  M. L. Samuels,et al.  Statistical Reversion Toward the Mean: More Universal Than Regression Toward the Mean , 1991 .

[6]  R. Clemen.  Combining forecasts: A review and annotated bibliography , 1989 .

[7]  A. H. Murphy,et al.  Scoring rules in probability assessment and evaluation , 1970 .

[8]  Lyle Brenner,et al.  Overconfidence in Probability and Frequency Judgments: A Critical Examination , 1996 .

[9]  Fergus Bolger,et al.  The calibration of subjective probability: Theories and models 1980–94. , 1994 .

[10]  Bruno CaprileIRST.  MODEL CALIBRATION , 1997 .

[11]  Peter Ayton,et al.  How Real is Overconfidence , 1997 .

[12]  Robert L. Winkler,et al.  Combining forecasts: A philosophical basis and some current issues , 1989 .

[13]  Rami Zwick,et al.  Comparing the calibration and coherence of numerical and verbal probability judgments , 1993 .

[14]  B. Fischhoff,et al.  Journal of Experimental Psychology: Human Learning and Memory , 1980 .

[15]  Ido Erev,et al.  On the Importance of Random Error in the Study of Probability Judgment. Part I: New Theoretical Developments , 1997 .

[16]  David V. Budescu,et al.  On the Importance of Random Error in the Study of Probability Judgment. Part II: Applying the Stochastic Judgment Model to Detect Systematic Trends , 1997 .

[17]  Christian Genest,et al.  Modeling Expert Judgments for Bayesian Updating , 1985 .

[18]  David V. Budescu,et al.  Decisions based on numerically and verbally expressed uncertainties. , 1988 .

[19]  Robert L. Winkler,et al.  Aggregating Point Estimates: A Flexible Modeling Approach , 1993 .

[20]  A. Tversky,et al.  Judgment under Uncertainty: Heuristics and Biases , 1974, Science.

[21]  Jack B. Soll,et al.  Overconfidence: It Depends on How, What, and Whom You Ask. , 1999, Organizational behavior and human decision processes.

[22]  Baruch Fischhoff,et al.  Calibration of Probabilities: The State of the Art , 1977 .

[23]  D. Budescu,et al.  Averaging probability judgments: Monte Carlo analyses of asymptotic diagnostic value , 2001 .

[24]  Nigel Harvey,et al.  Confidence in judgment , 1997, Trends in Cognitive Sciences.

[25]  P. Juslin,et al.  Naive empiricism and dogmatism in confidence research: a critical examination of the hard-easy effect. , 2000, Psychological review.

[26]  R. May,et al.  Overconfidence as a result of incomplete and wrong knowledge. , 1986 .

[27]  Phillip E. Pfeifer,et al.  Are We Overconfident in the Belief That Probability Forecasters Are Overconfident , 1994 .

[28]  Claudia González-Vallejo,et al.  Statement Verification: A Stochastic Model of Judgment and Response. , 1994 .

[29]  G. Gigerenzer,et al.  Probabilistic mental models: a Brunswikian theory of confidence. , 1991, Psychological review.

[30]  G. Brier,et al.  External correspondence: Decompositions of the mean probability score , 1982 .

[31]  D. Medin,et al.  Decision making from a cognitive perspective , 1995 .

[32]  B. Fischhoff,et al.  Reasons for confidence. , 1980 .

[33]  William R. Ferrell,et al.  Discrete subjective probabilities and decision analysis: Elicitation, calibration and combination. , 1994 .

[34]  Henrik Olsson,et al.  Naive empiricism and dogmatism in confidence research: a critical examination of the hard-easy effect. , 2000 .

[35]  Robin M. Hogarth,et al.  A note on aggregating opinions , 1978 .

[36]  Amnon Rapoport,et al.  Deterministic vs probabilistic strategies in detection , 1971 .

[37]  A. Diederich,et al.  Evaluating and Combining Subjective Probability Estimates , 1997 .

[38]  Adele Diederich,et al.  Understanding pooled subjective probability estimates , 2001, Math. Soc. Sci..

[39]  Ilan Yaniv,et al.  Measures of Discrimination Skill in Probabilistic Judgment , 1991 .

[40]  J V Baranski,et al.  Realism of confidence in sensory discrimination , 1999, Perception & psychophysics.

[41]  A. H. Murphy,et al.  Reliability of Subjective Probability Forecasts of Precipitation and Temperature , 1977 .

[42]  William R. Ferrell,et al.  Combining Individual Judgments , 1985 .

[43]  Jack B. Soll.  Determinants of Overconfidence and Miscalibration: The Roles of Random Error and Ecological Structure☆ , 1996 .

[44]  B. Fischhoff,et al.  Calibration of probabilities: the state of the art to 1980 , 1982 .

[45]  Anders Winman,et al.  The importance of item selection in ''knew-it-all-along'' studies of general knowledge , 1997 .

[46]  R. L. Winkler,et al.  Unanimity and compromise among probability forecasters , 1990 .

[47]  I. Erev,et al.  Simultaneous Over- and Underconfidence: The Role of Error in Judgment Processes. , 1994 .

[48]  J. Baranski,et al.  The calibration and resolution of confidence in perceptual judgments , 1994, Perception & psychophysics.

[49]  Huanping Dai,et al.  Signal Detection Analysis of the Ideal Group , 1994 .

[50]  N. Sanders,et al.  Journal of behavioral decision making: "The need for contextual and technical knowledge in judgmental forecasting", 5 (1992) 39-52 , 1992 .

[51]  J. Klayman.  Varieties of Confirmation Bias , 1995 .

[52]  D. Gilbert.  How mental systems believe. , 1991 .

[53]  Peter A. Morris,et al.  Combining Expert Judgments: A Bayesian Approach , 1977 .

[54]  Thomas S. Wallsten.  An Analysis of Judgment Research Analyses , 1996 .

[55]  I. Erev,et al.  The effect of explicit probabilities on decision weights and on the reflection effect , 1993 .

[56]  R. Clemen,et al.  Combining forecasts: A review and annotated bibliography , 1989 .

[57]  Roland W. Scholz,et al.  Current issues in West German decision research , 1986 .

[58]  William R. Ferrell,et al.  A model of calibration for subjective probabilities , 1980 .

[59]  P. Juslin,et al.  Realism of confidence in sensory discrimination: The underconfidence phenomenon , 1993, Perception & psychophysics.

[60]  C R Peterson,et al.  Intuitive inference about normally distributed populations. , 1968, Journal of experimental psychology.

[61]  Robert H. Ashton,et al.  Combining the judgments of experts: How many and which ones? , 1986 .