Measurement of subjective probability

Abstract Assessments of subjective probabilities do not always correspond with the assessor's true beliefs. It is therefore essential to provide him with an incentive to make honest assessments. This could be accomplished by using a scoring rule (a function of the assessed probabilities and the event which eventually occurs). The paper discusses various properties that scoring rules should possess in order to encourage honesty. Examples of scoring rules are given together with some practical and experimental experience with scoring rules. The paper also includes a survey of assessment techniques which do not rely on scoring rules.

[1]  R. L. Winkler.  Scoring Rules and the Evaluation of Probability Assessors , 1969 .

[2]  B. D. Finetti.  La prévision : ses lois logiques, ses sources subjectives , 1937 .

[3]  Carl-Axel S. Staël von Holstein,et al.  Two techniques for assessment of subjective probability distributions — An experimental study☆ , 1971 .

[4]  L. J. Savage.  Elicitation of Personal Probabilities and Expectations , 1971 .

[5]  W. Edwards,et al.  Conservatism in a simple probability inference task. , 1966, Journal of experimental psychology.

[6]  A. H. Murphy,et al.  “Good” Probability Assessors , 1968 .

[7]  R. L. Winkler.  The Assessment of Prior Distributions in Bayesian Analysis , 1967 .

[8]  Donald G. Morrison,et al.  Critique of: “Ranking Procedures and Subjective Probability Distributions” , 1967 .

[9]  S. Holstein,et al.  A Family of Strictly Proper Scoring Rules Which Are Sensitive to Distance , 1970 .

[10]  Joseph L. Zinnes,et al.  Theory and Methods of Scaling. , 1958 .

[11]  H. E. Massengill,et al.  AIRMAN QUALIFYING EXAMINATION-66 ADMINISTERED AS A CONFIDENCE TEST , 1968 .

[12]  L. J. Savage,et al.  The Foundations of Statistics , 1955 .

[13]  E. H. Shuford,et al.  Admissible probability measurement procedures , 1966, Psychometrika.

[14]  R. L. Winkler.  The Quantification of Judgment: Some Methodological Suggestions , 1967 .

[15]  B. deFinetti,et al.  METHODS FOR DISCRIMINATING LEVELS OF PARTIAL KNOWLEDGE CONCERNING A TEST ITEM. , 1965, The British journal of mathematical and statistical psychology.

[16]  C. Holstein.  An Experiment in Probabilistic Weather Forecasting , 1971 .

[17]  Edward S. Epstein,et al.  A Scoring System for Probability Forecasts of Ranked Categories , 1969 .

[18]  A. H. Murphy.  On the “Ranked Probability Score” , 1969 .

[19]  A. H. Murphy,et al.  A Note on the Utility of Probabilistic Predictions and the Probability Score in the Cost-Loss Ratio Decision Situation , 1966 .

[20]  J. Pfanzagl,et al.  Remarks on the measurement of subjective probability and information , 1967 .

[21]  S. Fienberg,et al.  Inference and Disputed Authorship: The Federalist , 1966 .

[22]  Masanao Toda,et al.  MEASUREMENT OF SUBJECTIVE PROBABILITY DISTRIBUTIONS , 1963 .

[23]  A. H. Murphy,et al.  Nonlinear Utility and the Probability Score , 1970 .

[24]  A. H. Murphy,et al.  THE RANKED PROBABILITY SCORE AND THE PROBABILITY SCORE: A COMPARISON , 1970 .