Scoring rules and the evaluation of probabilities

SummaryIn Bayesian inference and decision analysis, inferences and predictions are inherently probabilistic in nature. Scoring rules, which involve the computation of a score based on probability forecasts and what actually occurs, can be used to evaluate probabilities and to provide appropriate incentives for “good” probabilities. This paper review scoring rules and some related measures for evaluating probabilities, including decompositions of scoring rules and attributes of “goodness” of probabilites, comparability of scores, and the design of scoring rules for specific inferential and decision-making problems

[1]  B. D. Finetti.  La prévision : ses lois logiques, ses sources subjectives , 1937 .

[2]  G. Brier.  VERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITY , 1950 .

[3]  L. J. Savage,et al.  The Foundations of Statistics , 1955 .

[4]  J McCarthy,et al.  MEASURES OF THE VALUE OF INFORMATION. , 1956, Proceedings of the National Academy of Sciences of the United States of America.

[5]  D. Lindley.  On a Measure of the Information Provided by an Experiment , 1956 .

[6]  F. Sanders.  On Subjective Probability Forecasting , 1963 .

[7]  B. deFinetti,et al.  METHODS FOR DISCRIMINATING LEVELS OF PARTIAL KNOWLEDGE CONCERNING A TEST ITEM. , 1965, The British journal of mathematical and statistical psychology.

[8]  E. H. Shuford,et al.  Admissible probability measurement procedures , 1966, Psychometrika.

[9]  R. L. Winkler.  The Assessment of Prior Distributions in Bayesian Analysis , 1967 .

[10]  R. L. Winkler.  The Quantification of Judgment: Some Methodological Suggestions , 1967 .

[11]  A. H. Murphy,et al.  “Good” Probability Assessors , 1968 .

[12]  R. L. Winkler.  Scoring Rules and the Evaluation of Probability Assessors , 1969 .

[13]  Edward S. Epstein,et al.  A Scoring System for Probability Forecasts of Ranked Categories , 1969 .

[14]  A. H. Murphy,et al.  THE RANKED PROBABILITY SCORE AND THE PROBABILITY SCORE: A COMPARISON , 1970 .

[15]  S. Holstein,et al.  Assessment and evaluation of subjective probability distributions , 1970 .

[16]  L. J. Savage.  Elicitation of Personal Probabilities and Expectations , 1971 .

[17]  G. Hadley,et al.  Variational methods in economics , 1972 .

[18]  A. H. Murphy,et al.  Scalar and Vector Partitions of the Probability Score: Part I. Two-State Situation , 1972 .

[19]  A. H. Murphy,et al.  Hedging and Skill Scores for Probability Forecasts , 1973 .

[20]  A. H. Murphy.  A New Vector Partition of the Probability Score , 1973 .

[21]  A. H. Murphy,et al.  A Sample Skill Score for Probability Forecasts , 1974 .

[22]  Carl-Axel S. Staël von Holstein,et al.  Exceptional Paper---Probability Encoding in Decision Analysis , 1975 .

[23]  R. L. Winkler,et al.  Scoring Rules for Continuous Probability Distributions , 1976 .

[24]  A. H. Murphy,et al.  The Value of Climatological, Categorical and Probabilistic Forecasts in the Cost-Loss Ratio Situation , 1977 .

[25]  Judea Pearl,et al.  An economic basis for certain methods of evaluating probabilistic forecasts , 1978 .

[26]  Wayne S. Smith,et al.  Adaptive Forecasting Models Based on Predictive Distributions , 1978 .

[27]  Allan H. Murphy,et al.  The Family of Quadratic Scoring Rules , 1978 .

[28]  R. L. Keeney,et al.  Decisions with Multiple Objectives: Preferences and Value Trade-Offs , 1977, IEEE Transactions on Systems, Man, and Cybernetics.

[29]  J. Bernardo.  Expected Information as Expected Utility , 1979 .

[30]  Rakesh K. Sarin,et al.  Performance-Based Incentive Plans , 1980 .

[31]  Wayne S. Smith,et al.  Interactive Elicitation of Opinion for a Normal Linear Model , 1980 .

[32]  M. Degroot,et al.  Assessing Probability Assessors: Calibration and Refinement. , 1981 .

[33]  G. Brier,et al.  External correspondence: Decompositions of the mean probability score , 1982 .

[34]  D. V. Lindley.  [Scoring Rules and the Inevitability of Probability]: Reply to Discussion , 1982 .

[35]  A. H. Murphy,et al.  Assessing the Value of Frost Forecasts to Orchardists: A Dynamic Decision-Making Approach , 1982 .

[36]  A. Dawid.  The Well-Calibrated Bayesian , 1982 .

[37]  David Lindley.  Scoring rules and the inevitability of probability , 1982 .

[38]  Stephen E. Fienberg,et al.  The Comparison and Evaluation of Forecasters. , 1983 .

[39]  David V. Budescu,et al.  Encoding subjective probabilities: A psychological and psychometric review , 1983 .

[40]  A. H. Murphy,et al.  Impacts of Feedback and Experience on the Quality of Subjective Probability Forecasts. Comparison of Results from the First and Second Years of the Zierikzee Experiment , 1984 .

[41]  A. H. Murphy,et al.  Probability Forecasting in Meteorology , 1984 .

[42]  Shawn P. Curley,et al.  Conditional distribution analyses of probabilistic forecasts , 1985 .

[43]  G. Blattenberger,et al.  Separating the Brier Score into Calibration and Refinement Components: A Graphical Exposition , 1985 .

[44]  Robert L. Winkler,et al.  Expert resolution , 1986 .

[45]  J. Bernardo.  Approximations in Statistics from a Decision-Theoretical Viewpoint , 1987 .

[46]  A. H. Murphy,et al.  A General Framework for Forecast Verification , 1987 .

[47]  D. G. Rees,et al.  Foundations of Statistics , 1989 .

[48]  Rubin Herman,et al.  A WEAK SYSTEM OF AXIOMS FOR "RATIONAL" BEHAVIOR AND THE NONSEPARABILITY OF UTILITY FROM PRIOR , 1987 .

[49]  J. Frank Yates,et al.  Analyzing the accuracy of probability judgments for multiple events: An extension of the covariance decomposition , 1988 .

[50]  Methods in Economics , 1988 .

[51]  R. L. Winkler,et al.  Separating probability elicitation from utilities , 1988 .

[52]  G. Blattenberger,et al.  An Application of Operational-Subjective Statistical Methods to Rational Expectations , 1988 .

[53]  Ronald A. Howard,et al.  Readings on the Principles and Applications of Decision Analysis , 1989 .

[54]  M. Schervish.  A General Method for Comparing Probability Assessors , 1989 .

[55]  Max Henrion,et al.  Uncertainty: A Guide to Dealing with Uncertainty in Quantitative Risk and Policy Analysis , 1990 .

[56]  R. Zeckhauser,et al.  Principals and Agents: The Structure of Business , 1990 .

[57]  A. H. Murphy.  Forecast verification: Its Complexity and Dimensionality , 1991 .

[58]  Ralph L. Keeney,et al.  Eliciting probabilities from experts in complex technical problems , 1991 .

[59]  R. Cooke.  Experts in Uncertainty: Opinion and Subjective Probability in Science , 1991 .

[60]  Roman Krzysztofowicz.  Bayesian Correlation Score: A Utilitarian Measure of Forecast Skill , 1992 .

[61]  A. H. Murphy,et al.  Diagnostic verification of probability forecasts , 1992 .

[62]  J. Riley,et al.  The analytics of uncertainty and information: Long-run relationships and the credibility of threats and promises , 1992 .

[63]  M. L. Eaton.  A Statistical Diptych: Admissible Inferences--Recurrence of Symmetric Markov Chains , 1992 .

[64]  David J. Spiegelhalter,et al.  Bayesian analysis in expert systems , 1993 .

[65]  Peter C. Fishburn,et al.  Several Bayesians: A review , 1993 .

[66]  Thomas A. Louis,et al.  Graphical Elicitation of a Prior Distribution for a Clinical Trial , 1993 .

[67]  A. H. Murphy,et al.  What Is a Good Forecast? An Essay on the Nature of Goodness in Weather Forecasting , 1993 .

[68]  Robert L. Winkler,et al.  Evaluating and Combining Physicians' Probabilities of Survival in an Intensive Care Unit , 1993 .

[69]  T. Modis,et al.  Experts in uncertainty , 1993 .

[70]  John C. Harsanyi,et al.  Games with Incomplete Information , 1994 .

[71]  R. L. Winkler.  Evaluating probabilities: asymmetric scoring rules , 1994 .

[72]  James O. Berger,et al.  An overview of robust Bayesian analysis , 1994 .

[73]  R. L. Winkler,et al.  Coherent combination of experts' opinions , 1995 .

[74]  M. Schervish.  Theory of Statistics , 1995 .

[75]  Daniel S. Wilks,et al.  Statistical Methods in the Atmospheric Sciences: An Introduction , 1995 .

[76]  Allan H. Murphy.  A Coherent Method of Stratification within a General Framework for Forecast Verification , 1995 .

[77]  A. H. Murphy,et al.  General Decompositions of MSE-Based Skill Scores: Measures of Some Basic Aspects of Forecast Quality , 1996 .

[78]  G. Blattenberger.  Money demand revisited: An operational subjective approach , 1996 .

[79]  Deirdre N. McCloskey,et al.  The Standard Error of Regressions , 1996 .