On Setting Response Criteria for Calibrated Subjective Probability Estimates.

Framing the issue of subjective probability calibration in signal-detection-theory terms, this paper first proves a theorem regarding the placement of well-calibrated response criteria and then develops an algorithm guaranteed to find such criteria, should they exist. Application of this algorithm to tasks varying in difficulty and number of response categories shows that perfect calibration is easiest to attain under median difficulty levels (d' approximately 1.4) and is practically or theoretically impossible to attain when the task is either very hard (d' approximately 0.5) or very easy (d' approximately 10). Implications for calibration research, including the hard-easy effect, are discussed. Copyright 2001 Academic Press.

[1]  R. Luce,et al.  On the possible psychophysical laws. , 1959, Psychological review.

[2]  G. Gigerenzer,et al.  Probabilistic mental models: a Brunswikian theory of confidence. , 1991, Psychological review.

[3]  G. Keren.  Calibration and probability judgements: Conceptual and methodological issues , 1991 .

[4]  A. H. Murphy,et al.  A comparative evaluation of objective and subjective weather forecasts in the united states , 1984 .

[5]  Janet A. Sniezek,et al.  The effect of choosing on confidence in choice , 1990 .

[6]  David V. Budescu,et al.  On the Importance of Random Error in the Study of Probability Judgment. Part II: Applying the Stochastic Judgment Model to Detect Systematic Trends , 1997 .

[7]  B. Fischhoff,et al.  Reasons for confidence. , 1980 .

[8]  Claudia González-Vallejo,et al.  Statement Verification: A Stochastic Model of Judgment and Response. , 1994 .

[9]  B. Fischhoff,et al.  Calibration of probabilities: the state of the art to 1980 , 1982 .

[10]  David V. Budescu,et al.  Subjective estimation of precise and vague uncertainties. , 1987 .

[11]  A. Tversky,et al.  Judgment under Uncertainty: Heuristics and Biases , 1974, Science.

[12]  Juslin,et al.  The Calibration Issue: Theoretical Comments on Suantak, Bolger, and Ferrell (1996). , 1998, Organizational behavior and human decision processes.

[13]  D. Budescu,et al.  Preferences and reasons for communicating probabilistic information in verbal or numerical terms , 1993 .

[14]  William R. Ferrell,et al.  A model of calibration for subjective probabilities , 1980 .

[15]  G. Brier,et al.  External correspondence: Decompositions of the mean probability score , 1982 .

[16]  P. Juslin,et al.  Naive empiricism and dogmatism in confidence research: a critical examination of the hard-easy effect. , 2000, Psychological review.

[17]  W. Ferrell,et al.  The Hard-Easy Effect in Subjective Probability Calibration , 1996 .