Eliciting properties of probability distributions

We investigate the problem of truthfully eliciting an expert's assessment of a property of a probability distribution, where a property is any real-valued function of the distribution such as mean or variance. We show that not all properties are elicitable; for example, the mean is elicitable and the variance is not. For those that are elicitable, we provide a representation theorem characterizing all payment (or "score") functions that induce truthful revelation. We also consider the elicitation of sets of properties. We then observe that properties can always be inferred from sets of elicitable properties. This naturally suggests the concept of elicitation complexity; the elicitation complexity of property is the minimal size of such a set implying the property. Finally we discuss applications to prediction markets.

[1]  F. O'carroll.  Subjective Probabilities and Short‐Term Economic Forecasts: An Empirical Investigation , 1977 .

[2]  D J Spiegelhalter,et al.  Probabilistic prediction in patient management and clinical trials. , 1986, Statistics in medicine.

[3]  J. Wolfers,et al.  Prediction Markets , 2003 .

[4]  A. Hendrickson,et al.  Proper Scores for Probability Forecasters , 1971 .

[5]  A. H. Murphy,et al.  Probability Forecasting in Meteorology , 1984 .

[6]  David A. Bessler,et al.  Subjective Probabilities and Scoring Rules: Experimental Evidence , 1989 .

[7]  A. H. Murphy,et al.  Scoring rules and the evaluation of probabilities , 1996 .

[8]  A. Raftery,et al.  Strictly Proper Scoring Rules, Prediction, and Estimation , 2007 .

[9]  A. Buja,et al.  Loss Functions for Binary Class Probability Estimation and Classification: Structure and Applications , 2005 .

[10]  D. Friedman.  Effective Scoring Rules for Probabilistic Forecasts , 1983 .

[11]  Robert Forsythe,et al.  Anatomy of an Experimental Political Stock Market , 1992 .

[12]  M. Schervish.  A General Method for Comparing Probability Assessors , 1989 .

[13]  L. J. Savage.  Elicitation of Personal Probabilities and Expectations , 1971 .

[14]  Robin Hanson,et al.  Combinatorial Information Market Design , 2003, Inf. Syst. Frontiers.

[15]  G. Brier.  VERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITY , 1950 .

[16]  S. Holstein,et al.  A Family of Strictly Proper Scoring Rules Which Are Sensitive to Distance , 1970 .

[17]  Joan Feigenbaum,et al.  Computation in a distributed information market , 2003, EC '03.

[18]  E. H. Shuford,et al.  Admissible probability measurement procedures , 1966, Psychometrika.