A choice prediction competition: Choices from experience and from description

Erev, Ert, and Roth organized three choice prediction competitions focused on three related choice tasks: one shot decisions from description (decisions under risk), one shot decisions from experience, and repeated decisions from experience. Each competition was based on two experimental datasets: An estimation dataset, and a competition dataset. The studies that generated the two datasets used the same methods and subject pool, and examined decision problems randomly selected from the same distribution. After collecting the experimental data to be used for estimation, the organizers posted them on the Web, together with their fit with several baseline models, and challenged other researchers to compete to predict the results of the second (competition) set of experimental sessions. Fourteen teams responded to the challenge: the last seven authors of this paper are members of the winning teams. The results highlight the robustness of the difference between decisions from description and decisions from experience. The best predictions of decisions from descriptions were obtained with a stochastic variant of prospect theory assuming that the sensitivity to the weighted values decreases with the distance between the cumulative payoff functions. The best predictions of decisions from experience were obtained with models that assume reliance on small samples. Merits and limitations of the competition method are discussed.

[1]  R. Clemen.  Combining forecasts: A review and annotated bibliography , 1989 .

[2]  R. Hertwig,et al.  Decisions from Experience and the Effect of Rare Events in Risky Choice , 2004, Psychological science.

[3]  J. March,et al.  Adaptation as Information Restriction: The Hot Stove Effect , 2001 .

[4]  D. Prelec.  The Probability Weighting Function , 1998 .

[5]  B. Fischhoff,et al.  Judgment and decision making. , 2012, Wiley interdisciplinary reviews. Cognitive science.

[6]  E. Yechiam,et al.  Loss aversion, diminishing sensitivity, and the effect of experience on repeated decisions† , 2008 .

[7]  A. Sachs,et al.  Mathematical Cuneiform Texts. , 1946 .

[8]  M. Weber.  Methodology of Social Sciences , 1949 .

[9]  I. Erev,et al.  Experiment-Based Exams and the Difference Between the Behavioral and the Natural Sciences , 2005 .

[10]  M. Birnbaum,et al.  Testing Descriptive Utility Theories: Violations of Stochastic Dominance and Cumulative Independence , 1998 .

[11]  I. Erev,et al.  Learning, risk attitude and hot stoves in restless bandit problems , 2009 .

[12]  R. Hertwig,et al.  The priority heuristic: making choices without trade-offs. , 2006, Psychological review.

[13]  Ralph Hertwig,et al.  What impacts the impact of rare events , 2008 .

[14]  Deborah Monique,et al.  Authors' addresses , 2004 .

[15]  M. Allais.  Le comportement de l'homme rationnel devant le risque : critique des postulats et axiomes de l'ecole americaine , 1953 .

[16]  I. Erev,et al.  On adaptation, maximization, and reinforcement learning among cognitive strategies. , 2005, Psychological review.

[17]  Robert L. Slonim,et al.  Learning and equilibrium as useful approximations: Accuracy of prediction on randomly selected constant sum games , 2007 .

[18]  A. Tversky,et al.  Advances in prospect theory: Cumulative representation of uncertainty , 1992 .

[19]  T. C. Stewart,et al.  Equivalence: A novel basis for model comparison , 2007 .

[20]  Christian Lebiere,et al.  Competitive Modeling Symposium: PokerBot World Series , 2004, ICCM.

[21]  Robyn M. Dawes,et al.  The False Consensus Effect and Overconfidence: Flaws in Judgment or Flaws in How We Study Judgment? , 1996 .

[22]  R. Hertwig,et al.  The description–experience gap in risky choice , 2009, Trends in Cognitive Sciences.

[23]  R. Selten.  Axiomatic Characterization of the Quadratic Scoring Rule , 1998 .

[24]  I. Erev,et al.  Simultaneous Over- and Underconfidence: The Role of Error in Judgment Processes. , 1994 .

[25]  E. Weber,et al.  Predicting Risk-Sensitivity in Humans and Lower Animals: Risk as Variance or Coefficient of Variation , 2004, Psychological review.

[26]  J. Quiggin.  ON THE OPTIMAL DESIGN OF LOTTERIES , 1991 .

[27]  Colin Camerer,et al.  Experience‐weighted Attraction Learning in Normal Form Games , 1999 .

[28]  T. Evgeniou,et al.  To combine or not to combine: selecting among forecasts and their combinations , 2005 .

[29]  Cleotilde Gonzalez,et al.  Instance-based learning in dynamic decision making , 2003 .

[30]  G MarchJames,et al.  Adaptation as Information Restriction , 2001 .

[31]  S. S. Stevens.  On the psychophysical law. , 1957, Psychological review.

[32]  Rudolf Carnap,et al.  ON THE COMPARATIVE CONCEPT OF CONFIRMATION* , 1953, The British Journal for the Philosophy of Science.

[33]  A. Roth.  What Have We Learned from Market Design? , 2008 .

[34]  G. Brier.  VERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITY , 1950 .

[35]  I. Erev,et al.  Small feedback‐based decisions and their limited correspondence to description‐based decisions , 2003 .

[36]  Richard P. Larrick,et al.  Intuitions About Combining Opinions: Misappreciation of the Averaging Principle , 2006, Manag. Sci..

[37]  Timothy J. Pleskac,et al.  The game of life : how small samples render choice simpler , 2008 .

[38]  R. W. Sloley.  Mathematical Cuneiform Texts , 1946, Nature.

[39]  C. Lebiere,et al.  Stochastic Resonance in Human Cognition: ACT-R Versus Game Theory, Associative Neural Networks, Recursive Neural Networks, Q-Learning, and Humans , 2005 .

[40]  Alvin E. Roth,et al.  The effect of adding a constant to all payoffs: experimental investigation, and implications for reinforcement learning models , 1999 .

[41]  J. Rieskamp.  The probabilistic nature of preferential choice. , 2008, Journal of experimental psychology. Learning, memory, and cognition.

[42]  C. Lebiere,et al.  The Atomic Components of Thought , 1998 .

[43]  Mandeep K. Dhami,et al.  The role of representative design in an ecological approach to cognition. , 2004, Psychological bulletin.

[44]  A. Timmermann.  Chapter 4 Forecast Combinations , 2006 .

[45]  H. Simon,et al.  Studies of Scientific Discovery: Complementary Approaches and Convergent Findings , 1999 .

[46]  J. Busemeyer,et al.  Model Comparisons and Model Selections Based on Generalization Criterion Methodology. , 2000, Journal of mathematical psychology.

[47]  K. R. Hammond.  Human judgment and social policy , 1980 .

[48]  L. J. Savage,et al.  The Utility Analysis of Choices Involving Risk , 1948, Journal of Political Economy.

[49]  J. Busemeyer.  Decision making under uncertainty: a comparison of simple scalability, fixed-sample, and sequential-sampling models. , 1985, Journal of experimental psychology. Learning, memory, and cognition.

[50]  Timothy J. Pleskac,et al.  The game of life , 2008 .

[51]  N. Chater,et al.  Are Probabilities Overweighted or Underweighted When Rare Outcomes Are Experienced (Rarely)? , 2009, Psychological science.

[52]  Richard Gonzalez,et al.  Curvature of the Probability Weighting Function , 1996 .

[53]  J. Armstrong,et al.  PRINCIPLES OF FORECASTING 1 Principles of Forecasting : A Handbook for Researchers and Practitioners , 2006 .

[54]  R. Hogarth,et al.  Unit weighting schemes for decision making , 1975 .

[55]  Robert L. Slonim,et al.  Combining a Theoretical Prediction with Experimental Evidence , 2002 .

[56]  I. Erev,et al.  The Medium Prizes Paradox: Evidence From a Simulated Casino , 2001 .

[57]  John R. Anderson,et al.  Reflections of the Environment in Memory Form of the Memory Functions , 2022 .

[58]  B. Newell,et al.  Biased samples not mode of presentation: Re-examining the apparent underweighting of rare events in experience-based choice , 2008 .

[59]  Michael K. Martin,et al.  Instance-Based Decision Making Model of Repeated Binary Choice , 2007 .

[60]  Jasmina Arifovic,et al.  An initial implementation of the Turing tournament to learning in repeated two-person games , 2006, Games Econ. Behav..

[61]  A. Tversky,et al.  Prospect theory: an analysis of decision under risk — Source link , 2007 .

[62]  Timothy J. Pleskac,et al.  The Description-Experience Gap in Risky Choice: The Role of Sample Size and Experienced Probabilities , 2008 .

[63]  A. Tversky,et al.  Prospect theory: analysis of decision under risk , 1979 .

[64]  A. Timmermann.  Forecast Combinations , 2005 .