Measures of Discrimination Skill in Probabilistic Judgment

People's ability to assess probabilities of various events has been the topic of much interest in the areas of judgment, prediction, decision making, and memory. The evaluation of probabilistic judgments, however, raises interesting logical questions as to what it means to be a"good" judge. In this article, we focus on a normative concept of probabilistic accuracy that we call discrimination and present a measure of a judge's discrimination skill. This measure builds on an earlier index (Murphy, 1973) and has the advantages that (a) it can be interpreted as the percentage of variance accounted for by the judge and (b) it is unbiased. By way of deriving this new discrimination measure, we also show that it is related to Pearson's chi-square statistic, a result which may be useful in the future development of hypothesis testing and estimation procedures.

[1]  L. J. Savage,et al.  The Foundations of Statistics , 1955 .

[2]  B. Fischhoff,et al.  Knowing with Certainty: The Appropriateness of Extreme Confidence. , 1977 .

[3]  R. Hastie,et al.  Hindsight: Biased judgments of past events after the outcomes are known. , 1990 .

[4]  B. Fischhoff,et al.  Journal of Experimental Psychology: Human Learning and Memory , 1980 .

[5]  G. W. Snedecor.  STATISTICAL METHODS , 1967 .

[6]  Stephen J. Hoch,et al.  Outcome feedback: hindsight and information , 1989 .

[7]  Shawn P. Curley,et al.  Conditional distribution analyses of probabilistic forecasts , 1985 .

[8]  Baruch Fischhoff,et al.  Calibration of Probabilities: The State of the Art , 1977 .

[9]  Robert L. Winkler,et al.  Probabilistic Prediction: Some Experimental Results , 1971 .

[10]  Ilan Yaniv,et al.  A case study of expert judgment: Economists' probabilities versus base-rate model forecasts , 1992 .

[11]  A. H. Murphy.  A New Vector Partition of the Probability Score , 1973 .

[12]  Steven D. Penrod,et al.  Performance feedback improves the resolution of confidence judgments , 1988 .

[13]  N. Sanders,et al.  Journal of behavioral decision making: "The need for contextual and technical knowledge in judgmental forecasting", 5 (1992) 39-52 , 1992 .

[14]  A. Tversky,et al.  Judgment under Uncertainty: Heuristics and Biases , 1974, Science.

[15]  B. Fischhoff,et al.  Calibration of probabilities: the state of the art to 1980 , 1982 .

[16]  G. Brier.  VERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITY , 1950 .

[17]  G. Brier,et al.  External correspondence: Decompositions of the mean probability score , 1982 .

[18]  B. Fischhoff,et al.  Reasons for confidence. , 1980 .

[19]  Stephen J. Hoch,et al.  Counterfactual reasoning and accuracy in predicting personal events. , 1985 .

[20]  D. C. Howell.  Statistical Methods for Psychology , 1987 .

[21]  Arthur M. Glenberg,et al.  Calibration of comprehension. , 1985 .

[22]  David V. Budescu,et al.  Encoding subjective probabilities: A psychological and psychometric review , 1983 .

[23]  David Dunning,et al.  The overconfidence effect in social prediction. , 1990 .

[24]  J. I. Kim,et al.  Accountability and judgment processes in a personality prediction task. , 1987, Journal of personality and social psychology.

[25]  A. H. Murphy,et al.  “Good” Probability Assessors , 1968 .