Psychometric modeling of decision making via game play

We build a model for the kind of decision making involved in games of strategy such as chess, making it abstract enough to remove essentially all game-specific contingency, and compare it to known psychometric models of test taking, item response, and performance assessment. Decisions are modeled in terms of fallible agents Z faced with possible actions ai whose utilities Ui=U (ai) are not fully apparent. The three main goals of the model are prediction, meaning to infer probabilities Pi for Z to choose ai; intrinsic rating, meaning to assess the skill of a person's actual choices ait over various test items t; and simulation of the distribution of choices by an agent with a specified skill set. We describe and train the model on large data from chess tournament games of different ranks of players, and exemplify its accuracy by applying it to give intrinsic ratings for world championship matches.

[1]  Graeme Hutcheson,et al.  Rasch Models for Measurement , 2011 .

[2]  Richard E. Korf,et al.  Generalized Game Trees , 1989, IJCAI.

[3]  E. B. Andersen,et al.  CONDITIONAL INFERENCE FOR MULTIPLE‐CHOICE QUESTIONNAIRES , 1973 .

[4]  G. Masters.  A rasch model for partial credit scoring , 1982 .

[5]  G. Rasch.  On General Laws and the Meaning of Measurement in Psychology , 1961 .

[6]  Christopher F. Chabris,et al.  Visualization, pattern recognition, and forward search: effects of playing speed and sight of the position on grandmaster chess errors , 2003, Cogn. Sci..

[7]  Giuseppe Di Fatta,et al.  Performance and Prediction: Bayesian Modelling of Fallible Choice in Chess , 2009, ACG.

[8]  F A Wichmann,et al.  Ning for Helpful Comments and Suggestions. This Paper Benefited Con- Siderably from Conscientious Peer Review, and We Thank Our Reviewers the Psychometric Function: I. Fitting, Sampling, and Goodness of Fit , 2001 .

[9]  Guy Haworth.  Gentlemen, Stop Your Engines! , 2007, J. Int. Comput. Games Assoc..

[10]  Guy Haworth,et al.  Intrinsic Chess Ratings , 2011, AAAI.

[11]  Guy Haworth.  Reference Fallible Endgame Play , 2003, J. Int. Comput. Games Assoc..

[12]  Ivan Bratko,et al.  Computer Analysis of World Chess Champions , 2006, J. Int. Comput. Games Assoc..

[13]  Bruce W. Ballard,et al.  Non-Minimax Search Strategies for Use Against Fallible Opponents , 1983, AAAI.

[14]  Giuseppe Di Fatta,et al.  Skill rating by Bayesian inference , 2009, 2009 IEEE Symposium on Computational Intelligence and Data Mining.

[15]  M. Glickman.  Parameter Estimation in Large Dynamic Paired Comparison Experiments , 1999 .

[16]  A. Elo.  The rating of chessplayers, past and present , 1978 .

[17]  Tom Minka,et al.  TrueSkill Through Time: Revisiting the History of Chess , 2007, NIPS.

[18]  Mark Levene,et al.  A Discrete Evolutionary Model for Chess Players' Ratings , 2011, IEEE Transactions on Computational Intelligence and AI in Games.

[19]  D. Andrich.  A rating formulation for ordered response categories , 1978 .

[20]  Andrej Favia,et al.  Data Analysis Using Item Response Theor y Methodology: An Introduction to Selected Programs and Applications. , 2012 .

[21]  Peter J. Jansen.  KQKR: Awareness of a Fallible Opponent , 1992, J. Int. Comput. Games Assoc..

[22]  John M Linacre,et al.  Rasch analysis of rank-ordered data. , 2006, Journal of applied measurement.

[23]  Stephen D. Baker,et al.  Testing the test: Item response curves and test quality , 2006 .

[24]  Georg Rasch,et al.  Probabilistic Models for Some Intelligence and Attainment Tests , 1981, The SAGE Encyclopedia of Research Design.

[25]  Paul Abelson.  Stop your engines , 2005 .

[26]  E. Wagenmakers,et al.  A psychometric analysis of chess expertise. , 2005, The American journal of psychology.

[27]  F. Baker.  The basics of item response theory , 1985 .

[28]  Guy Haworth,et al.  Understanding Distributions of Chess Performances , 2011, ACG.

[29]  Jerad H. Moxley,et al.  The role of intuition and deliberative thinking in experts’ superior tactical decision-making , 2012, Cognition.

[30]  Ivan Bratko,et al.  HOW TRUSTWORTHY IS CRAFTY’S ANALYSIS OF WORLD CHESS CHAMPIONS? , 2008 .

[31]  Ivan Bratko,et al.  Using Heuristic-Search Based Engines for Estimating Human Skill at Chess , 2011, J. Int. Comput. Games Assoc..

[32]  Remo Ostini,et al.  Polytomous Item Response Theory Models , 2005 .