A Subjectivist View of Calibration.

Abstract : Calibration concerns the relationship between subjective probabilities and the long-run frequencies of events. Theorems from the statistical and probability literature are reviewed to discover the conditions under which a coherent Bayesian expects to be calibrated. If the probability assessor knows the outcomes of all previous events when making each assessment, calibration is always expected. However, when such outcome feedback is lacking, the assessor expects to be well calibrated on an exchangeable set of events if and only if the events in question are viewed as independent. Although this strong condition has not been tested in previous research, we speculate that the past findings of pervasive overconfidence are not invalid. Although experimental studies of calibration hold promise for the development of cognitive theories of confidence, their value for the practice of probability assessment seems more limited. Efforts to train probability assessors to be calibrated may be misplaced. (Author)

[1]  L. J. Savage,et al.  The Foundations of Statistics , 1955 .

[2]  B. Fischhoff,et al.  Knowing with Certainty: The Appropriateness of Extreme Confidence. , 1977 .

[3]  Peter A. Morris,et al.  Decision Analysis Expert Use , 1974 .

[4]  Lee B. Lusted,et al.  Introduction to medical decision making , 1968 .

[5]  B. Fischhoff,et al.  Calibration of probabilities: the state of the art to 1980 , 1982 .

[6]  Stuart Oskamp,et al.  The relationship of clinical experience and training methods to several criteria of clinical prediction. , 1962 .

[7]  Peter A. Morris,et al.  Combining Expert Judgments: A Bayesian Approach , 1977 .

[8]  Rex V. Brown,et al.  Decision Analysis for the Manager , 1974 .

[9]  E. H. Shuford,et al.  Admissible probability measurement procedures , 1966, Psychometrika.

[10]  B. Fischhoff,et al.  Reasons for confidence. , 1980 .

[11]  A. Tversky,et al.  Judgment under Uncertainty: Heuristics and Biases , 1974, Science.

[12]  William Feller,et al.  An Introduction to Probability Theory and Its Applications , 1951 .

[13]  Michel Loève,et al.  Probability Theory I , 1977 .

[14]  B. Fischhoff,et al.  Training for calibration. , 1980 .

[15]  Abner Shimony,et al.  Coherence and the axioms of confirmation , 1955, Journal of Symbolic Logic.

[16]  P. A. Adams,et al.  Training in confidence-judgments. , 1958, The American journal of psychology.

[17]  William R. Ferrell,et al.  A model of calibration for subjective probabilities , 1980 .

[18]  John G. Kemeny,et al.  Fair bets and inductive probabilities , 1955, Journal of Symbolic Logic.

[19]  R. Sherman Lehman,et al.  On confirmation and rational betting , 1955, Journal of Symbolic Logic.

[20]  H. Kyburg,et al.  Studies in Subjective Probability (2nd ed). , 1981 .

[21]  F. Reif,et al.  Fundamentals of Statistical and Thermal Physics , 1965 .