A penny for your thoughts: a survey of methods for eliciting beliefs

Incentivized methods for eliciting subjective probabilities in economic experiments present the subject with risky choices that encourage truthful reporting. We discuss the most prominent elicitation methods and their underlying assumptions, provide theoretical comparisons and give a new justification for the quadratic scoring rule. On the empirical side, we survey the performance of these elicitation methods in actual experiments, considering also practical issues of implementation such as order effects, hedging, and different ways of presenting probabilities and payment schemes to experimental subjects. We end with a discussion of the trade-offs involved in using incentives for belief elicitation and some guidelines for implementation.

[1]  Asa B. Palley,et al.  Lossed in translation: an off-the-shelf method to recover probabilistic beliefs from loss-averse agents , 2015, Experimental economics.

[2]  A. Schotter,et al.  Belief Elicitation in the Laboratory , 2014 .

[3]  G. Harrison,et al.  Estimating subjective probabilities , 2014, Journal of Risk and Uncertainty.

[4]  K. Schlag,et al.  Simple Belief Elicitation , 2014 .

[5]  G. Harrison,et al.  Eliciting Subjective Probabilities with Binary Lotteries , 2014 .

[6]  Glenn W. Harrison,et al.  Inducing risk neutral preferences with binary lotteries: A reconsideration , 2013 .

[7]  Alexander Smith,et al.  Estimating the causal effect of beliefs on contributions in repeated public good games , 2013 .

[8]  Tanjim Hossain,et al.  The Binarized Scoring Rule , 2013 .

[9]  K. Schlag,et al.  UvA-DARE ( Digital Academic Repository ) Eliciting probabilities , means , medians , variances and covariances without assuming risk neutrality , 2013 .

[10]  C. Noussair,et al.  Information aggregation and belief elicitation in experimental parimutuel betting markets , 2012 .

[11]  Daniel Houser,et al.  Belief elicitation in the presence of naïve respondents: An experimental study , 2012 .

[12]  Gijs van de Kuilen,et al.  Belief Elicitation: A Horse Race Among Truth Serums , 2011 .

[13]  Stephanie W. Wang Incentive effects: The case of belief elicitation from individuals in groups , 2011 .

[14]  Glenn W. Harrison,et al.  What do prediction markets predict? , 2011 .

[15]  J. Vergnaud,et al.  Subjective beliefs formation and elicitation rules : experimental evidence , 2010 .

[16]  Simon Gächter,et al.  The effects of (incentivized) belief elicitation in public goods experiments , 2010 .

[17]  P. Wakker,et al.  Comonotonic proper scoring rules to measure ambiguity and subjective beliefs , 2010 .

[18]  Olivier Armantier,et al.  Eliciting beliefs: Proper scoring rules, incentives, stakes and hedging , 2010 .

[19]  Florian Artinger,et al.  Applying Quadratic Scoring Rule transparently in multiple choice settings: A note , 2010 .

[20]  Georg Weizsäcker,et al.  Beliefs and Actions in the Trust Game: Creating Instrumental Variables to Estimate the Causal Effect , 2010, Games Econ. Behav..

[21]  E. Elisabet Rutström,et al.  Stated beliefs versus inferred beliefs: A methodological inquiry and experimental test , 2009, Games Econ. Behav..

[22]  Nicolas Treich,et al.  Subjective Probabilities in Games: An Application to the Overbidding Puzzle , 2009 .

[23]  J. Sonnemans,et al.  A Truth Serum for Non-Bayesians: Correcting Proper Scoring Rules for Risk Attitudes* , 2009 .

[24]  Robert L. Winkler,et al.  Evaluating Quantile Assessments , 2009, Oper. Res..

[25]  Thomas R. Palfrey,et al.  On Eliciting Beliefs in Strategic Games , 2009 .

[26]  G. Harrison,et al.  Expected utility theory and prospect theory: one wedding and a decent funeral , 2009 .

[27]  Edi Karni,et al.  A Mechanism for Eliciting Probabilities , 2009 .

[28]  David J. McKenzie,et al.  Measuring Subjective Expectations in Developing Countries: A Critical Review and New Evidence , 2009 .

[29]  Yoav Shoham,et al.  Eliciting properties of probability distributions: the highlights , 2008, SECO.

[30]  Yoav Shoham,et al.  Eliciting properties of probability distributions , 2008, EC '08.

[31]  Daniel John Zizzo,et al.  Experimenter demand effects in economic experiments , 2008 .

[32]  Alexander K. Koch,et al.  Belief elicitation in experiments: is there a hedging problem? , 2008 .

[33]  A. Raftery,et al.  Strictly Proper Scoring Rules, Prediction, and Estimation , 2007 .

[34]  Paul Resnick,et al.  Eliciting Informative Feedback: The Peer-Prediction Method , 2005, Manag. Sci..

[35]  A. O'Hagan,et al.  Statistical Methods for Eliciting Probability Distributions , 2005 .

[36]  J. Shogren,et al.  An Experimental Comparison of Induced and Elicited Beliefs , 2005 .

[37]  Peter Ockenfels,et al.  Measuring Strategic Uncertainty in Coordination Games , 2004, SSRN Electronic Journal.

[38]  D. Prelec A Bayesian Truth Serum for Subjective Data , 2004, Science.

[39]  C. Manski Interpreting the Predictions of Prediction Markets , 2004 .

[40]  A. Schotter,et al.  An Experimental Study of Belief Learning Using Elicited Beliefs , 2002 .

[41]  C. Manski Identification of decision rules in experiments on simple games of proposal and response , 2002 .

[42]  Gerardo A. Guerra,et al.  Trust Responsiveness and Beliefs , 2002 .

[43]  Charles A. Holt,et al.  Risk Aversion and Incentive Effects , 2002 .

[44]  Steffen Huck,et al.  Do players correctly estimate what others do?: Evidence of conservatism in beliefs , 2002 .

[45]  Haiqin Wang,et al.  A method for evaluating elicitation schemes for probabilistic models , 2002, IEEE Trans. Syst. Man Cybern. Part B.

[46]  B. Rimer,et al.  General Performance on a Numeracy Scale among Highly Educated Samples , 2001, Medical decision making : an international journal of the Society for Medical Decision Making.

[47]  F. Ramsey The Foundations of Mathematics and Other Logical Essays , 2001 .

[48]  Dirk Engelmann,et al.  The False Consensus Effect Disappears if Representative Information and Monetary Incentives Are Given , 2000 .

[49]  Rachel Croson,et al.  Thinking like a game theorist: factors affecting the frequency of equilibrium play , 2000 .

[50]  R. Selten,et al.  Money Does Not Induce Risk Neutral Behavior, but Binary Lotteries Do even Worse , 1999 .

[51]  Colin Camerer,et al.  The Effects of Financial Incentives in Experiments: A Review and Capital-Labor-Production Framework , 1999 .

[52]  J. Jaffray,et al.  Elicitation of Subjective Probabilities when the Initial Endowment is Unobservable , 1999 .

[53]  David Kjellberg Measuring Expectations , 1999, Identification Problems in the Social Sciences.

[54]  P. C. Price,et al.  Effects of a Relative-Frequency Elicitation Question on Likelihood Judgment Accuracy: The Case of External Correspondence. , 1998, Organizational behavior and human decision processes.

[55]  R. Selten Axiomatic Characterization of the Quadratic Scoring Rule , 1998 .

[56]  Dean P. Foster,et al.  Precision and Accuracy of Judgmental Estimation , 1997 .

[57]  J. Sonnemans,et al.  Value Orientations, Expectations and Voluntary Contributions in Public Goods. , 1996 .

[58]  A. H. Murphy,et al.  Scoring rules and the evaluation of probabilities , 1996 .

[59]  Dean P. Foster,et al.  Graininess of judgment under uncertainty: An accuracy-informativeness trade-off , 1995 .

[60]  Gerd Gigerenzer,et al.  How to Improve Bayesian Reasoning Without Instruction: Frequency Formats , 1995 .

[61]  Zvi Safra,et al.  The impossibility of experimental elicitation of subjective probabilities , 1995 .

[62]  A. Tversky,et al.  Support theory: A nonextensional representation of subjective probability. , 1994 .

[63]  D. Önkal,et al.  An evaluation of the reliability of probability judgments across response modes and over time , 1993 .

[64]  Ido Erev,et al.  The Negative Effect of Probability Assessments on Decision Quality , 1993 .

[65]  D. Schmeidler,et al.  A More Robust Definition of Subjective Probability , 1992 .

[66]  R. McKelvey,et al.  Public and private information : an experimental study of information pooling , 1990 .

[67]  B. M. Hill,et al.  Theory of Probability , 1990 .

[68]  M. Schervish A General Method for Comparing Probability Assessors , 1989 .

[69]  R. L. Winkler,et al.  Separating probability elicitation from utilities , 1988 .

[70]  Franklin Allen Notes---Discovering Personal Probabilities When Utility Functions are Unknown , 1987 .

[71]  D. Friedman Effective Scoring Rules for Probabilistic Forecasts , 1983 .

[72]  G. W. Fischer,et al.  Scoring-rule feedback and the overconfidence syndrome in subjective probability forecasting , 1982 .

[73]  D. M. Grether,et al.  Financial Incentive Effects and Individual Decision-making , 1981 .

[74]  A. Tversky,et al.  On the Reconciliation of Probability Assessments , 1979 .

[75]  R. L. Winkler,et al.  Scoring Rules for Continuous Probability Distributions , 1976 .

[76]  D. V. Gokhale,et al.  Theory of Probability, Vol. I , 1975 .

[77]  D. Fryback,et al.  Relative Variance Preferences in a Choice-Among-Bets Paradigm , 1973 .

[78]  Michael L Donnell,et al.  Intrasubject comparison of four response modes for “subjective probability” assessment , 1973 .

[79]  Charles Vlek,et al.  Coherence of human judgment in a limited probabilistic environment , 1973 .

[80]  C. Vlek THE FAIR BETTING GAME AS AN ADMISSIBLE PROCEDURE FOR ASSESSMENT OF SUBJECTIVE PROBABILITIES , 1973 .

[81]  F. A. Jensen,et al.  Psychological effects of proper scoring rules , 1973 .

[82]  L. J. Savage Elicitation of Personal Probabilities and Expectations , 1971 .

[83]  A. H. Murphy,et al.  Nonlinear Utility and the Probability Score , 1970 .

[84]  A. H. Murphy,et al.  “Good” Probability Assessors , 1968 .

[85]  L D Phillips,et al.  Subjective propabilities inferred from estimates and bets. , 1967, Journal of experimental psychology.

[86]  David A. Schum,et al.  Subjective probability revisions under several cost-payoff arrangements , 1967 .

[87]  W. Edwards,et al.  Conservatism in a simple probability inference task. , 1966, Journal of experimental psychology.

[88]  E. H. Shuford,et al.  Admissible probability measurement procedures , 1966, Psychometrika.

[89]  B. deFinetti,et al.  METHODS FOR DISCRIMINATING LEVELS OF PARTIAL KNOWLEDGE CONCERNING A TEST ITEM. , 1965, The British journal of mathematical and statistical psychology.

[90]  Thornton B Roby,et al.  BELIEF STATES: A PRELIMINARY EMPIRICAL STUDY , 1964 .

[91]  Masanao Toda,et al.  MEASUREMENT OF SUBJECTIVE PROBABILITY DISTRIBUTIONS , 1963 .

[92]  J McCarthy,et al.  MEASURES OF THE VALUE OF INFORMATION. , 1956, Proceedings of the National Academy of Sciences of the United States of America.

[93]  L. J. Savage,et al.  The Foundation of Statistics , 1956 .

[94]  F. Ramsey Truth and Probability , 2016 .

[95]  Timo Hoffmann The Effect of Belief Elicitation Game Play , 2014 .

[96]  M. Krawczyk Overconfident for real? Proper scoring for confidence intervals , 2011 .

[97]  K. Schlag,et al.  A Penny for your Thoughts: Incentives for Interval Elicitation. , 2011 .

[98]  Thomas A. Rietz,et al.  Chapter 115 On the Performance of the Lottery Procedure for Controlling Risk Preferences , 2008 .

[99]  J. Shogren,et al.  Belief Elicitation: An Experimental Comparison of Scoring Rule and Prediction Methods , 2007 .

[100]  David Jenkinson,et al.  The Elicitation of Probabilities-A Review of the Statistical Literature , 2005 .

[101]  K. Vind A foundation for statistics , 2003 .

[102]  Theo Offerman,et al.  Is the Quadratic Scoring Rule Behaviorally Incentive Compatible , 2001 .

[103]  Nick Feltovich,et al.  Thinking Like a Game Theorist: Comment , 2000 .

[104]  G. G. Allen,et al.  Risk-risk analysis , 1994, Journal of Risk and Uncertainty.

[105]  I. Gilboa Expected utility with purely subjective non-additive probabilities , 1987 .

[106]  Robyn M. Dawes,et al.  Behavior, communication, and assumptions about other people's behavior in a commons dilemma situation. , 1977 .

[107]  Richard Schmalensee,et al.  An Experimental Study of Expectation Formation , 1976 .

[108]  Bruno de Finetti,et al.  Logical foundations and measurement of subjective probability , 1970 .

[109]  Carl-Axel S. Staël von Holstein,et al.  Measurement of subjective probability , 1970 .

[110]  James A. Wise,et al.  Subjective probability revision and subsequent decisions , 1969 .

[111]  M. Degroot,et al.  Measuring utility by a single-response sequential method. , 1964, Behavioral science.

[112]  Cedric A. B. Smith,et al.  Consistency in Statistical Inference and Decision , 1961 .

[113]  Masanao Toda,et al.  MEASUREMEIT OF INTUITIVE-PROBABILITY BY A METHOD OF GAME , 1951 .

[114]  G. Brier VERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITY , 1950 .

[115]  L. M. M.-T. Theory of Probability , 1929, Nature.