A random support model of the calibration of subjective probabilities

Abstract A stochastic model of the calibration of subjective probabilities based on support theory ( Rottenstreich and Tversky, 1997 , Tversky and Koehler, 1994 ) is presented. This model extends support theory—a general representation of probability judgment—to the domain of calibration, the analysis of the correspondence between subjective and objective probability. The random support model can account for the common finding of overconfidence, and also predicts the form of the relationship between overconfidence and item difficulty (the “hard–easy effect”). The parameters of the model have natural psychological interpretations, such as discriminability between correct and incorrect hypotheses, and extremity of judgment. The random support model can be distinguished from other stochastic models of calibration by: (a) using fewer parameters, (b) eliminating the use of variable cutoffs by mapping underlying support directly into judged probability, (c) allowing validation of model parameters with independent assessments of support, and (d) applying to a wide variety of tasks by framing probability judgment in the integrative context of support theory.

[1]  Claudia González-Vallejo,et al.  Statement Verification: A Stochastic Model of Judgment and Response. , 1994 .

[2]  Derek J. Koehler,et al.  Heuristics and Biases: Remarks on Support Theory: Recent Advances and Future Directions , 2002 .

[3]  W. Ferrell,et al.  A model for realism of confidence judgments: Implications for underconfidence in sensory discrimination , 1995, Perception & psychophysics.

[4]  I Erev,et al.  The importance of theory: response to Brenner (2000) , 2000, Psychological review.

[5]  W. Ferrell.  Calibration of sensory and cognitive judgments: a single model for both. , 1994, Scandinavian journal of psychology.

[6]  L. Thurstone.  A law of comparative judgment. , 1994 .

[7]  L. Brenner,et al.  Should observed overconfidence be dismissed as a statistical artifact? Critique of Erev, Wallsten, and Budescu (1994) , 2000, Psychological review.

[8]  William R. Ferrell,et al.  The Effect of Base Rate on Calibration of Subjective Probability for True-False Questions: Model and Experiment , 1983 .

[9]  B. Mellers,et al.  Similarity and Choice. , 1994 .

[10]  A. Tversky,et al.  Heuristics and Biases: Unpacking, Repacking, and Anchoring: Advances in Support Theory , 2002 .

[11]  David V. Budescu,et al.  Encoding subjective probabilities: A psychological and psychometric review , 1983 .

[12]  Jack B. Soll,et al.  Overconfidence: It Depends on How, What, and Whom You Ask. , 1999, Organizational behavior and human decision processes.

[13]  G. Gigerenzer,et al.  Probabilistic mental models: a Brunswikian theory of confidence. , 1991, Psychological review.

[14]  P. Juslin.  The Overconfidence Phenomenon as a Consequence of Informal Experimenter-Guided Selection of Almanac Items , 1994 .

[15]  C. Gettys,et al.  MINERVA-DM: A memory processes model for judgments of likelihood. , 1999 .

[16]  Howard E. Egeth,et al.  Conceptual and perceptual components of interletter inhibition. , 1981 .

[17]  David V. Budescu,et al.  On the Importance of Random Error in the Study of Probability Judgment. Part II: Applying the Stochastic Judgment Model to Detect Systematic Trends , 1997 .

[18]  Derek J. Koehler,et al.  The enhancement effect in probability judgment , 1997 .

[19]  C. Fox.  Strength of Evidence, Judged Probability, and Choice Under Uncertainty , 1999, Cognitive Psychology.

[20]  D. Kahneman,et al.  Heuristics and Biases: The Psychology of Intuitive Judgment , 2002 .

[21]  J. Frank Yates,et al.  Subjective probability accuracy analysis. , 1994 .

[22]  M. Björkman,et al.  Knowledge, calibration, and resolution: A linear model , 1992 .

[23]  Elizabeth L Bjork,et al.  On the nature of input channels in visual processing. , 1977, Psychological review.

[24]  L. Brenner,et al.  Similarity between Hypotheses and Evidence , 1999, Cognitive Psychology.

[25]  M. Björkman.  Internal Cue Theory: Calibration and Resolution of Confidence in General Knowledge , 1994 .

[26]  Lyle Brenner,et al.  Overconfidence in Probability and Frequency Judgments: A Critical Examination , 1996 .

[27]  Ayleen Wisudha,et al.  Distribution of probability assessments for almanac and future event questions , 1982 .

[28]  B. Fischhoff,et al.  Calibration of probabilities: the state of the art to 1980 , 1982 .

[29]  G. Keren.  Calibration and probability judgements: Conceptual and methodological issues , 1991 .

[30]  G. Keren,et al.  On the ability of monitoring non-veridical perceptions and uncertain knowledge: some calibration studies. , 1988, Acta psychologica.

[31]  P. Juslin,et al.  Thurstonian and Brunswikian origins of uncertainty in judgment: a sampling model of confidence in sensory discrimination. , 1997, Psychological review.

[32]  Derek J. Koehler,et al.  Subjective Probability of Disjunctive Hypotheses: Local-Weight Models for Decomposition of Evidential Support , 1999, Cognitive Psychology.

[33]  Jack B. Soll.  Determinants of Overconfidence and Miscalibration: The Roles of Random Error and Ecological Structure☆ , 1996 .

[34]  David L. Ronis,et al.  Components of probability judgment accuracy: Individual consistency and effects of subject matter and assessment method. , 1987 .

[35]  William R. Ferrell,et al.  Discrete subjective probabilities and decision analysis: Elicitation, calibration and combination. , 1994 .

[36]  A. Tversky,et al.  Judgment under Uncertainty: Heuristics and Biases , 1974, Science.

[37]  Derek J. Koehler,et al.  Heuristics and Biases: The Calibration of Expert Judgment: Heuristics and Biases Beyond the Laboratory , 2002 .

[38]  Carl-Axel S. Staël von Holstein,et al.  Exceptional Paper---Probability Encoding in Decision Analysis , 1975 .

[39]  A. Tversky,et al.  The weighing of evidence and the determinants of confidence , 1992, Cognitive Psychology.

[40]  B. Fischhoff,et al.  Subjective Confidence in Forecasts. , 1982 .

[41]  William R. Ferrell,et al.  A model of calibration for subjective probabilities , 1980 .

[42]  Ab McClelland,et al.  The calibration of subjective probabilities: Theories and models 1980-1993 , 1994 .

[43]  P. Juslin,et al.  Naive empiricism and dogmatism in confidence research: a critical examination of the hard-easy effect. , 2000, Psychological review.

[44]  W. Ferrell,et al.  The Hard-Easy Effect in Subjective Probability Calibration , 1996 .

[45]  A. Tversky,et al.  Support theory: A nonextensional representation of subjective probability. , 1994 .

[46]  I. Erev,et al.  Simultaneous Over- and Underconfidence: The Role of Error in Judgment Processes. , 1994 .

[47]  Thomas S. Wallsten.  An Analysis of Judgment Research Analyses , 1996 .

[48]  H Gu,et al.  The effects of averaging subjective probability estimates between and within judges. , 2000, Journal of experimental psychology. Applied.

[49]  Derek J. Koehler,et al.  A Strength Model of Probability Judgments for Tournaments , 1996 .

[50]  E. Brunswik.  Representative design and probabilistic theory in a functional psychology. , 1955, Psychological review.

[51]  E. Brunswik.  Organismic achievement and environmental probability. , 1943 .

[52]  Phillip E. Pfeifer,et al.  Are We Overconfident in the Belief That Probability Forecasters Are Overconfident , 1994 .

[53]  Ido Erev,et al.  On the Importance of Random Error in the Study of Probability Judgment. Part I: New Theoretical Developments , 1997 .