A Continuous Information Gain Measure to Find the Most Discriminatory Problems for AI Benchmarking

This paper introduces an information-theoretic method for selecting a subset of problems which gives the most information about a group of problem-solving algorithms. This method was tested on the games in the General Video Game AI (GVGAI) framework, allowing us to identify a smaller set of games that still gives a large amount of information about the abilities of different game-playing agents. This approach can be used to make agent testing more efficient. We can achieve almost as good discriminatory accuracy when testing on only a handful of games as when testing on more than a hundred games, something which is often computationally infeasible. Furthermore, this method can be extended to study the dimensions of the effective variance in game design between these games, allowing us to identify which games differentiate between agents in the most complementary ways.

[1]  Wojciech Zaremba,et al.  OpenAI Gym , 2016, ArXiv.

[2]  P ? ? ? ? ? ? ? % ? ? ? ? , 1991 .

[3]  Julian Togelius,et al.  Matching Games and Algorithms for General Video Game Playing , 2021, AIIDE.

[4]  Mark J. Nelson,et al.  Investigating vanilla MCTS scaling on the GVG-AI game corpus , 2016, 2016 IEEE Conference on Computational Intelligence and Games (CIG).

[5]  Risto Miikkulainen,et al.  General Video Game Playing , 2013, Artificial and Computational Intelligence in Games.

[6]  Julian Togelius,et al.  Towards generating arcade game rules with VGDL , 2015, 2015 IEEE Conference on Computational Intelligence and Games (CIG).

[7]  José Hernández-Orallo,et al.  Dual Indicators to Analyze AI Benchmarks: Difficulty, Discrimination, Ability, and Generality , 2020, IEEE Transactions on Games.

[8]  Julian Togelius,et al.  Analyzing the robustness of general video game playing agents , 2016, 2016 IEEE Conference on Computational Intelligence and Games (CIG).

[9]  J. Carpenter.  May the best analyst win. , 2011, Science.

[10]  Julian Togelius,et al.  Hyper-heuristic general video game playing , 2016, 2016 IEEE Conference on Computational Intelligence and Games (CIG).

[11]  Tsuyoshi Murata,et al.  {m , 1934, ACML.

[12]  J. Hernández-Orallo,et al.  AI results for the Atari 2600 games : difficulty and discrimination using IRT , 2017 .

[13]  Simon M. Lucas,et al.  General Video Game for 2 players: Framework and competition , 2016, 2016 8th Computer Science and Electronic Engineering (CEEC).

[14]  Michael R. Genesereth,et al.  General Game Playing: Overview of the AAAI Competition , 2005, AI Mag..

[15]  Adolfo Martínez Usó,et al.  Item response theory in AI: Analysing machine learning classifiers at the instance level , 2019, Artif. Intell..

[16]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[17]  Julian Togelius,et al.  Deceptive Games , 2018, EvoApplications.

[18]  Julian Togelius,et al.  General Video Game AI: Competition, Challenges and Opportunities , 2016, AAAI.

[19]  Dennis J. N. J. Soemers,et al.  Enhancements for real-time Monte-Carlo Tree Search in General Video Game Playing , 2016, 2016 IEEE Conference on Computational Intelligence and Games (CIG).

[21]  P. Mahalanobis.  On the generalized distance in statistics , 1936 .

[22]  Julian Togelius,et al.  Artificial and Computational Intelligence in Games , 2013, Artificial and Computational Intelligence in Games.

[23]  Julian Togelius,et al.  Towards a Video Game Description Language , 2013, Artificial and Computational Intelligence in Games.

[24]  Julian Togelius,et al.  Ieee Transactions on Computational Intelligence and Ai in Games the 2014 General Video Game Playing Competition , 2022 .

[25]  Marc G. Bellemare,et al.  The Arcade Learning Environment: An Evaluation Platform for General Agents , 2012, J. Artif. Intell. Res..

[26]  Thomas M. Cover,et al.  Elements of Information Theory (Wiley Series in Telecommunications and Signal Processing) , 2006 .

[27]  Simon M. Lucas,et al.  Rolling horizon evolution enhancements in general video game playing , 2017, 2017 IEEE Conference on Computational Intelligence and Games (CIG).

[28]  Michael L. Littman,et al.  Bandit-Based Planning and Learning in Continuous-Action Markov Decision Processes , 2012, ICAPS.

[29]  Julian Togelius,et al.  General Video Game AI: A Multitrack Framework for Evaluating Agents, Games, and Content Generation Algorithms , 2018, IEEE Transactions on Games.