Learning and equilibrium as useful approximations: Accuracy of prediction on randomly selected constant sum games

There is a good deal of miscommunication among experimenters and theorists about how to evaluate a theory that can be rejected by sufficient data, but may nevertheless be a useful approximation. A standard experimental design reports whether a general theory can be rejected on an informative test case. This paper, in contrast, reports an experiment designed to meaningfully pose the question: “how good an approximation does a theory provide on average.” It focuses on a class of randomly selected games, and estimates how many pairs of experimental subjects would have to be observed playing a previously unexamined game before the mean of the experimental observations would provide a better prediction than the theory about the behavior of a new pair of subjects playing this game. We call this quantity the model’s equivalent number of observations, and explore its properties.

[1]  J. Neumann.  Zur Theorie der Gesellschaftsspiele , 1928 .

[2]  J. Nash.  Equilibrium Points in N-Person Games. , 1950, Proceedings of the National Academy of Sciences of the United States of America.

[3]  M. Friedman.  Essays in Positive Economics , 1954 .

[4]  M. Allais.  Le comportement de l'homme rationnel devant le risque : critique des postulats et axiomes de l'ecole americaine , 1953 .

[5]  Frederick Mosteller,et al.  Stochastic Models for Learning , 1956 .

[6]  R. Duncan Luce,et al.  Individual Choice Behavior , 1959 .

[7]  W. Edwards,et al.  Probability learning in 1000 trials. , 1961, Journal of experimental psychology.

[8]  Patrick Suppes,et al.  Markov learning models for multiperson interactions , 1962 .

[9]  Klaus Krickeberg,et al.  Markov learning models for multiperson interactions , 1962 .

[10]  Sidney Siegel,et al.  Choice, strategy, and utility , 1967 .

[11]  A. Roth,et al.  Game-Theoretic Models and the Role of Information in Bargaining , 1979 .

[12]  C. Granger,et al.  Improved methods of combining forecasts , 1984 .

[13]  B. O'Neill.  Nonmetric test of the minimax theory of two-person zerosum games. , 1987, Proceedings of the National Academy of Sciences of the United States of America.

[14]  Sunil Gupta,et al.  Combination of Forecasts: An Extension , 1987 .

[15]  Robert W. Rosenthal,et al.  Testing the Minimax Hypothesis: A Re-examination of O'Neill's Game Experiment , 1990 .

[16]  Barry O'Neill,et al.  Comments on Brown and Rosenthal's Reexamination [Testing the Minimax Hypothesis, A Reexamination of O'Neill's Game Experiment] , 1991 .

[17]  Jacob Cohen.  The earth is round (p < .05) , 1994 .

[18]  W. Estes.  Toward a Statistical Theory of Learning. , 1994 .

[19]  A. Roth,et al.  Learning in Extensive-Form Games: Experimental Data and Simple Dynamic Models in the Intermediate Term* , 1995 .

[20]  J. Kagel,et al.  Handbook of Experimental Economics , 1997 .

[21]  J. Ochs.  Games with Unique, Mixed Strategy Equilibria: An Experimental Study , 1995 .

[22]  Eyal Winter,et al.  Experimental study of repeated team-games , 1996 .

[23]  Mark Perlman,et al.  The Rational Foundations of Economic Behaviour , 1996 .

[24]  Daniel Friedman,et al.  Individual Learning in Normal Form Games: Some Laboratory Results☆☆☆ , 1997 .

[25]  John H. Kagel,et al.  Signalling and Adaptive Learning in an Entry Limit Pricing Game , 1997 .

[26]  Nagel,et al.  Experimental Results on the Centipede Game in Normal Form: An Investigation on Learning. , 1998, Journal of mathematical psychology.

[27]  A. Roth,et al.  Predicting How People Play Games: Reinforcement Learning in Experimental Games with Unique, Mixed Strategy Equilibria , 1998 .

[28]  A. Roth,et al.  An experimental study of sequential bargaining , 1998 .

[29]  A. Rapoport,et al.  Coordination, “Magic,” and Reinforcement Learning in a Market Entry Game , 1998 .

[30]  Ido Erev.  Signal detection by human observers: a cutoff reinforcement learning model of categorization decisions under uncertainty. , 1998 .

[31]  D. Fudenberg,et al.  The Theory of Learning in Games , 1998 .

[32]  Colin Camerer,et al.  Experience‐weighted Attraction Learning in Normal Form Games , 1999 .

[33]  J K Goeree,et al.  Stochastic game theory: for playing games, not just for doing theory. , 1999, Proceedings of the National Academy of Sciences of the United States of America.

[34]  E. Hopkins.  Two Competing Models of How People Learn in Games (first version) , 1999 .

[35]  Alvin E. Roth,et al.  The effect of adding a constant to all payoffs: experimental investigation, and implications for reinforcement learning models , 1999 .

[36]  Nick Feltovich,et al.  Reinforcement-based vs. Belief-based Learning Models in Experimental Asymmetric-information Games , 2000 .

[37]  Gary E. Bolton,et al.  ERC: A Theory of Equity, Reciprocity, and Competition , 2000 .

[38]  J. Busemeyer,et al.  Model Comparisons and Model Selections Based on Generalization Criterion Methodology. , 2000, Journal of mathematical psychology.

[39]  Ido Erev,et al.  On the potential uses and current limitations of data driven learning models , 2000 .

[40]  Jim Engle-Warnick,et al.  Inferring repeated-game strategies from actions: evidence from trust game experiments , 2006 .

[41]  John Wooders,et al.  On the Irrelevance of Risk Attitudes in Repeated Two-Outcome Games , 2001, Games Econ. Behav..

[42]  Farshid Vahid,et al.  Predicting How People Play Games: A Simple Dynamic Model of Choice , 2001, Games Econ. Behav..

[43]  Timothy C. Salmon.  An Evaluation of Econometric Models of Adaptive Learning , 2001 .

[44]  Alvin E. Roth,et al.  Predictive value and the usefulness of game theoretic models , 2002 .

[45]  A. Roth.  The Economist as Engineer: Game Theory, Experimentation, and Computation as Tools for Design Economics , 2002 .

[46]  Jörgen W. Weibull,et al.  Testing Game Theory , 2004 .

[47]  Werner Güth,et al.  Advances in understanding strategic behaviour : game theory, experiments and bounded rationality : essays in honour of Werner Güth , 2004 .

[48]  I. Erev,et al.  On adaptation, maximization, and reinforcement learning among cognitive strategies. , 2005, Psychological review.

[49]  Ido Erev,et al.  Generality, repetition, and the role of descriptive learning models , 2005 .

[50]  A. Roth,et al.  The Speed of Learning in Noisy Games: Partial Reinforcement and the Sustainability of Cooperation , 2006 .

[51]  Ido Erev,et al.  Replicated alternatives and the role of confusion, chasing, and regret in decisions from experience , 2007 .

[52]  C. Plott.  The Rational Foundations of Economic Behavior , 2008 .

[53]  R. Selten,et al.  Stationary Concepts for Experimental 2x2 Games , 2008 .