Measuring Level-K Reasoning, Satisficing, and Human Error in Game-Play Data

Inferences about structured patterns in human decision making have been drawn from medium-scale simulated competitions with human subjects. The concepts analyzed in these studies include level-k thinking, satisficing, and other human error tendencies. These concepts can be mapped via a natural depth of search metric into the domain of chess, where copious data is available from hundreds of thousands of games by players of a wide range of precisely known skill levels in real competitions. The games are analyzed by strong chess programs to produce authoritative utility values for move decision options by progressive deepening of search. Our experiments show a significant relationship between the formulations of level-k thinking and the skill level of players. Notably, the players are distinguished solely on moves where they erred -- according to the average depth level at which their errors are exposed by the authoritative analysis. Our results also indicate that the decisions are often independent of tail assumptions on higher-order beliefs. Further, we observe changes in this relationship in different contexts, such as minimal versus acute time pressure. We try to relate satisficing to insufficient level of reasoning and answer numerically the question, why do humans blunder?

[1]  A. Tversky,et al.  Weighing Risk and Uncertainty , 1995 .

[2]  Paul Abelson.  Stop your engines , 2005 .

[3]  Ivan Bratko,et al.  HOW TRUSTWORTHY IS CRAFTY’S ANALYSIS OF WORLD CHESS CHAMPIONS? , 2008 .

[4]  E. Glover.  DICTIONARY OF PSYCHOLOGY , 1959 .

[5]  Diogo R. Ferreira,et al.  The Impact of the Search Depth on Chess Playing Strength , 2013, J. Int. Comput. Games Assoc..

[6]  Kenneth W. Regan,et al.  Quantifying Depth and Complexity of Thinking and Knowledge , 2015, ICAART.

[7]  A. Tversky,et al.  Prospect theory: analysis of decision under risk , 1979 .

[8]  D. Kahneman.  A perspective on judgment and choice: mapping bounded rationality. , 2003, The American psychologist.

[9]  Ivan Bratko,et al.  Computer Analysis of World Chess Champions , 2006, J. Int. Comput. Games Assoc..

[10]  A. Tversky,et al.  Subjective Probability: A Judgment of Representativeness , 1972 .

[11]  A. Tversky,et al.  Prospect theory: an analysis of decision under risk — Source link , 2007 .

[12]  M. Kocher,et al.  Bargaining under time pressure in an experimental ultimatum game , 2003 .

[13]  Guy Haworth.  Gentlemen, Stop Your Engines! , 2007, J. Int. Comput. Games Assoc..

[14]  D. Stahl,et al.  Experimental evidence on players' models of other players , 1994 .

[15]  A. Damasio,et al.  Insensitivity to future consequences following damage to human prefrontal cortex , 1994, Cognition.

[16]  Guy Haworth,et al.  Intrinsic Chess Ratings , 2011, AAAI.

[17]  L. V. Allis,et al.  Searching for solutions in games and artificial intelligence , 1994 .

[18]  A. Tversky,et al.  Advances in prospect theory: Cumulative representation of uncertainty , 1992 .

[19]  A. Elo.  The rating of chessplayers, past and present , 1978 .

[20]  A. Damasio,et al.  Deciding Advantageously Before Knowing the Advantageous Strategy , 1997, Science.

[21]  Christopher F. Chabris,et al.  Visualization, pattern recognition, and forward search: effects of playing speed and sight of the position on grandmaster chess errors , 2003, Cogn. Sci..

[22]  Tomasz Strzalecki,et al.  Depth of Reasoning and Higher Order Beliefs , 2010 .

[23]  Matthias Sutter,et al.  Time is Money - Time Pressure, Incentives, and the Quality of Decision-Making , 2004 .

[24]  Kenneth W. Regan,et al.  Human and Computer Preferences at Chess , 2014, MPREF@AAAI.

[25]  H. Simon,et al.  Rational choice and the structure of the environment. , 1956, Psychological review.

[26]  Guy Haworth.  Reference Fallible Endgame Play , 2003, J. Int. Comput. Games Assoc..

[27]  A. Tversky,et al.  On the psychology of prediction , 1973 .

[28]  Matthias Sutter,et al.  Level-k reasoning and time pressure in the 11-20 money request game , 2013 .

[29]  Diogo R. Ferreira,et al.  Determining the Strength of Chess Players Based on Actual Play , 2012, J. Int. Comput. Games Assoc..

[30]  Amos J. Storkey,et al.  Training Deep Convolutional Neural Networks to Play Go , 2015, ICML.

[31]  Tamal Biswas,et al.  Designing Intelligent Agents to Judge Intrinsic Quality of Human Decisions , 2015, ICAART.

[32]  Ivan Bratko,et al.  Using Heuristic-Search Based Engines for Estimating Human Skill at Chess , 2011, J. Int. Comput. Games Assoc..

[33]  Ayala Arad,et al.  Multi-dimensional iterative reasoning in action: The case of the Colonel Blotto game , 2012 .

[34]  Amir Ban.  Automatic Learning of Evaluation, with Applications to Computer Chess , 2012 .

[35]  Carl-Axel S. Staël von Holstein,et al.  The Concept of Probability in Psychological Experiments , 1974 .

[36]  E. Wagenmakers,et al.  A psychometric analysis of chess expertise. , 2005, The American journal of psychology.

[37]  D. Stahl,et al.  On Players' Models of Other Players: Theory and Experimental Evidence , 1995 .

[38]  Kenneth W. Regan,et al.  Psychometric modeling of decision making via game play , 2013, 2013 IEEE Conference on Computational Inteligence in Games (CIG).

[39]  A. Rubinstein,et al.  The 11-20 Money Request Game: A Level-k Reasoning Study , 2012 .

[40]  Steffen Huck,et al.  Do players correctly estimate what others do?: Evidence of conservatism in beliefs , 2002 .

[41]  A. Tversky,et al.  Choices, Values, and Frames , 2000 .

[42]  Guy Haworth,et al.  Understanding Distributions of Chess Performances , 2011, ACG.