Navigating the landscape of multiplayer games

Multiplayer games have long been used as testbeds in artificial intelligence research, aptly referred to as the Drosophila of artificial intelligence. Traditionally, researchers have focused on using well-known games to build strong agents. This progress, however, can be better informed by characterizing games and their topological landscape. Tackling this latter question can facilitate understanding of agents and help determine what game an agent should target next as part of its training. Here, we show how network measures applied to response graphs of large-scale games enable the creation of a landscape of games, quantifying relationships between games of varying sizes and characteristics. We illustrate our findings in domains ranging from canonical games to complex empirical games capturing the performance of trained agents pitted against one another. Our results culminate in a demonstration leveraging this information to generate new and interesting games, including mixtures of empirical games synthesized from real world games.

[1]  Csaba Szepesvári,et al.  Bounds and dynamics for empirical game theoretic analysis , 2019, Autonomous Agents and Multi-Agent Systems.

[2]  J M Smith,et al.  Evolution and the theory of games , 1976 .

[3]  Sebastian Deterding,et al.  The Lens of Intrinsic Skill Atoms: A Method for Gameful Design , 2015, Hum. Comput. Interact..

[4]  M. Nowak,et al.  Stochastic dynamics of invasion and fixation. , 2006, Physical review. E, Statistical, nonlinear, and soft matter physics.

[5]  Kristinn R. Thórisson,et al.  Why Artificial Intelligence Needs a Task Theory - And What It Might Look Like , 2016, AGI.

[6]  M. Nowak,et al.  Evolutionary Dynamics of Biological Games , 2004, Science.

[7]  Josef Hofbauer,et al.  Evolutionary Games and Population Dynamics , 1998 .

[8]  John Scott.  What is social network analysis , 2010 .

[9]  L. da F. Costa,et al.  Characterization of complex networks: A survey of measurements , 2005, cond-mat/0505185.

[10]  Gillian Smith,et al.  Analyzing the expressive range of a level generator , 2010, PCGames@FDG.

[11]  Joe Marks,et al.  Automatic Design of Balanced Board Games , 2007, AIIDE.

[12]  A. Lesne.  Complex Networks: from Graph Theory to Biology , 2006 .

[13]  L. Vygotsky.  Interaction between learning and development , 1978 .

[14]  Julian Togelius,et al.  Automatic generation and analysis of physics-based puzzle games , 2013, 2013 IEEE Conference on Computational Inteligence in Games (CIG).

[15]  Vincent Conitzer,et al.  A double oracle algorithm for zero-sum security games on graphs , 2011, AAMAS.

[16]  Neil Burch,et al.  Heads-up limit hold’em poker is solved , 2015, Science.

[17]  Stanley Wasserman,et al.  Social Network Analysis: Methods and Applications , 1994 .

[18]  Julian Togelius,et al.  Towards Automatic Personalized Content Generation for Platform Games , 2010, AIIDE.

[19]  Jakub Kowalski,et al.  Evolving Chess-like Games Using Relative Algorithm Performance Profiles , 2016, EvoApplications.

[20]  L. Imhof,et al.  Stochasticity and evolutionary stability. , 2006, Physical review. E, Statistical, nonlinear, and soft matter physics.

[21]  José Hernández-Orallo,et al.  Evaluation in artificial intelligence: from task-oriented to ability-oriented measurement , 2017, Artificial Intelligence Review.

[22]  Jonathan Schaeffer,et al.  A Gamut of Games , 2001, AI Mag..

[23]  Terence D. Sanger,et al.  Neural network learning control of robot manipulators using gradually increasing task difficulty , 1994, IEEE Trans. Robotics Autom..

[24]  Michael Mateas,et al.  Answer Set Programming for Procedural Content Generation: A Design Space Approach , 2011, IEEE Transactions on Computational Intelligence and AI in Games.

[25]  Peter McBurney,et al.  An evolutionary game-theoretic comparison of two double-auction market designs , 2004, AAMAS'04.

[26]  José Hernández-Orallo,et al.  The Measure of All Minds: Evaluating Natural and Artificial Intelligence , 2017 .

[27]  Peter A. Flach,et al.  A Unified View of Performance Metrics: Translating Threshold Choice into Expected Classification Loss C` Esar Ferri , 2012 .

[28]  Simon Colton,et al.  The ANGELINA Videogame Design System—Part II , 2017, IEEE Transactions on Computational Intelligence and AI in Games.

[29]  Emmanuel Müller,et al.  NetLSD: Hearing the Shape of a Graph , 2018, KDD.

[30]  Julian Togelius,et al.  Mech-Elites: Illuminating the Mechanic Space of GVG-AI , 2020, FDG.

[31]  Marlos C. Machado,et al.  Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents (Extended Abstract) , 2018, IJCAI.

[32]  Sriram Srinivasan,et al.  OpenSpiel: A Framework for Reinforcement Learning in Games , 2019, ArXiv.

[33]  Joel Z. Leibo,et al.  Autocurricula and the Emergence of Innovation from Social Interaction: A Manifesto for Multi-Agent Intelligence Research , 2019, ArXiv.

[34]  Kevin Waugh,et al.  Solving Games with Functional Regret Estimation , 2014, AAAI Workshop: Computer Poker and Imperfect Information.

[35]  Rahul Savani,et al.  Robust Market Making via Adversarial Reinforcement Learning , 2020, IJCAI.

[36]  Max Jaderberg,et al.  Real World Games Look Like Spinning Tops , 2020, NeurIPS.

[37]  Alexiei Dingli,et al.  Platformer level design for player believability , 2016, 2016 IEEE Conference on Computational Intelligence and Games (CIG).

[38]  Max Jaderberg,et al.  Open-ended Learning in Symmetric Zero-sum Games , 2019, ICML.

[39]  Herbert Gintis,et al.  Game Theory Evolving: A Problem-Centered Introduction to Modeling Strategic Interaction - Second Edition , 2009 .

[40]  Carl Veller,et al.  Finite-population evolution with rare mutations in asymmetric games , 2015, J. Econ. Theory.

[41]  Alex Graves,et al.  Automated Curriculum Learning for Neural Networks , 2017, ICML.

[42]  Julian Togelius,et al.  Search-Based Procedural Content Generation: A Taxonomy and Survey , 2011, IEEE Transactions on Computational Intelligence and AI in Games.

[43]  Igor Mordatch,et al.  Emergent Tool Use From Multi-Agent Autocurricula , 2019, ICLR.

[44]  Wojciech M. Czarnecki,et al.  Grandmaster level in StarCraft II using multi-agent reinforcement learning , 2019, Nature.

[45]  Danai Koutra,et al.  NetSimile: A Scalable Approach to Size-Independent Network Similarity , 2012, ArXiv.

[46]  George Sugihara,et al.  Complex systems: Ecology for bankers , 2008, Nature.

[47]  Kevin Leyton-Brown,et al.  A Formal Separation Between Strategic and Nonstrategic Behavior , 2018, EC.

[48]  Maarten van Steen,et al.  Graph Theory and Complex Networks: An Introduction , 2010 .

[49]  Murray Campbell,et al.  Deep Blue , 2002, Artif. Intell..

[50]  Andrew Byde.  Applying evolutionary game theory to auction mechanism design , 2003, EC '03.

[51]  Xiaotie Deng,et al.  Settling the complexity of computing two-player Nash equilibria , 2007, JACM.

[52]  Xiaofeng Wang,et al.  Reinforcement Learning to Play an Optimal Nash Equilibrium in Team Markov Games , 2002, NIPS.

[53]  G. Tesauro,et al.  Analyzing Complex Strategic Interactions in Multi-Agent Systems , 2002 .

[54]  Julian Togelius,et al.  An experiment in automatic game design , 2008, 2008 IEEE Symposium On Computational Intelligence and Games.

[55]  Shimon Whiteson,et al.  Multi-Agent Common Knowledge Reinforcement Learning , 2018, NeurIPS.

[56]  R. Munos,et al.  Multiagent Evaluation under Incomplete Information , 2019, NeurIPS.

[57]  G. Hardin,et al.  The Tragedy of the Commons , 1968, Green Planet Blues.

[58]  Daniel Hennes,et al.  Neural Replicator Dynamics: Multiagent Learning via Hedging Policy Gradients , 2020, AAMAS.

[59]  Julian Togelius,et al.  Intentional computational level design , 2019, GECCO.

[60]  J. Robinson.  AN ITERATIVE METHOD OF SOLVING A GAME , 1951, Classics in Game Theory.

[61]  Thore Graepel,et al.  The Mechanics of n-Player Differentiable Games , 2018, ICML.

[62]  C. Hauert,et al.  Via Freedom to Coercion: The Emergence of Costly Punishment , 2007, Science.

[63]  Wim B. G. Liebrand,et al.  A Classification of Social Dilemma Games , 1983 .

[64]  N. Heess,et al.  A Generalized Training Approach for Multiagent Learning , 2019, ICLR.

[65]  Jacob Feldman,et al.  Minimization of Boolean complexity in human concept learning , 2000, Nature.

[66]  Julian Togelius,et al.  Illuminating Generalization in Deep Reinforcement Learning through Procedural Level Generation , 2018, 1806.10729.

[67]  Srinivasan Parthasarathy,et al.  Symmetrizations for clustering directed graphs , 2011, EDBT/ICDT '11.

[68]  J. McCarthy.  AI as Sport , 1997, Science.

[69]  Lior Rokach,et al.  Clustering Methods , 2005, The Data Mining and Knowledge Discovery Handbook.

[70]  Simon Colton,et al.  The ANGELINA Videogame Design System—Part I , 2017, IEEE Transactions on Computational Intelligence and AI in Games.

[71]  ORF Capture-Seq as a versatile method for targeted identification of full-length isoforms , 2020, Nature Communications.

[72]  B. Mohar.  THE LAPLACIAN SPECTRUM OF GRAPHS y , 1991 .

[73]  S. Risi,et al.  Increasing generality in machine learning through procedural content generation , 2019, Nature Machine Intelligence.

[74]  Michael P. Wellman,et al.  Analyzing Incentives for Protocol Compliance in Complex Domains: A Case Study of Introduction-Based Routing , 2013, ArXiv.

[75]  K. Sigmund.  The Calculus of Selfishness , 2010 .

[76]  Debora Donato,et al.  Large scale properties of the Webgraph , 2004 .

[77]  Joel Z. Leibo,et al.  A Generalised Method for Empirical Game Theoretic Analysis , 2018, AAMAS.

[78]  William V. Wright,et al.  A Theory of Fun for Game Design , 2004 .

[79]  Julian Togelius,et al.  General Video Game AI: A Multitrack Framework for Evaluating Agents, Games, and Content Generation Algorithms , 2018, IEEE Transactions on Games.

[80]  D. Bonchev,et al.  Complexity in chemistry, biology, and ecology , 2005 .

[81]  Martin A. Nowak,et al.  Evolving cooperation in multichannel games , 2020, Nature Communications.

[82]  Michael H. Bowling,et al.  Solving Imperfect Information Games Using Decomposition , 2013, AAAI.

[83]  D. Fudenberg,et al.  Evolutionary cycles of cooperation and defection. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[84]  Matthias Dehmer.  Structural Analysis of Complex Networks , 2010 .

[85]  Demis Hassabis,et al.  Mastering the game of Go with deep neural networks and tree search , 2016, Nature.

[86]  Bryan Randolph Bruns,et al.  Names for Games: Locating 2 × 2 Games , 2015, Games.

[87]  Michalis Vazirgiannis,et al.  Clustering and Community Detection in Directed Networks: A Survey , 2013, ArXiv.

[88]  J. Nash.  Equilibrium Points in N-Person Games. , 1950, Proceedings of the National Academy of Sciences of the United States of America.

[89]  Robert Tibshirani,et al.  Estimating the number of clusters in a data set via the gap statistic , 2000 .

[90]  Guy Lever,et al.  Emergent Coordination Through Competition , 2019, ICLR.

[91]  Paul W. Goldberg,et al.  The Complexity of Computing a Nash Equilibrium , 2009, SIAM J. Comput..

[92]  Iyad Rahwan,et al.  Cooperating with machines , 2017, Nature Communications.

[93]  David Silver,et al.  Fictitious Self-Play in Extensive-Form Games , 2015, ICML.

[94]  Jeff Clune,et al.  AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence , 2019, ArXiv.

[95]  Craig Boutilier,et al.  Regret-based Reward Elicitation for Markov Decision Processes , 2009, UAI.

[96]  Jörgen W. Weibull,et al.  Evolutionary Game Theory , 1996 .

[97]  Jason Weston,et al.  Curriculum learning , 2009, ICML '09.

[98]  Characteristics of Generatable Games , 2014 .

[99]  Joel Lehman,et al.  Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutions , 2020, ICML.

[100]  G. Caldarelli,et al.  Economic complexity: Conceptual grounding of a new metrics for global competitiveness , 2013 .

[101]  Nikolaus Hansen,et al.  The CMA Evolution Strategy: A Comparing Review , 2006, Towards a New Evolutionary Computation.

[102]  Pieter Abbeel,et al.  Reverse Curriculum Generation for Reinforcement Learning , 2017, CoRL.

[103]  Drew Fudenberg,et al.  Imitation Processes with Small Mutations , 2004, J. Econ. Theory.

[104]  Simon Colton,et al.  Multi-faceted evolution of simple arcade games , 2011, 2011 IEEE Conference on Computational Intelligence and Games (CIG'11).

[105]  Georgios N. Yannakakis,et al.  Evolving opponents for interesting interactive computer games , 2004 .

[106]  D. Pham,et al.  Selection of K in K-means clustering , 2005 .

[107]  Mark W. Youngblood,et al.  Author Correction: Integrated genomic analyses of de novo pathways underlying atypical meningiomas , 2018, Nature Communications.

[108]  Reinhard Schneider,et al.  Using graph theory to analyze biological networks , 2011, BioData Mining.

[109]  Dima Shepelyansky,et al.  Spectral properties of the Google matrix of the World Wide Web and other directed networks , 2010, Physical review. E, Statistical, nonlinear, and soft matter physics.

[110]  Yishay Mansour,et al.  Nash Convergence of Gradient Dynamics in General-Sum Games , 2000, UAI.

[111]  Michael P. Wellman.  Methods for Empirical Game-Theoretic Analysis , 2006, AAAI.

[112]  Kristinn R. Thórisson,et al.  Task Analysis for Teaching Cumulative Learners , 2018, AGI.

[113]  F. C. Santos,et al.  Stochastic Dynamics through Hierarchically Embedded Markov Chains. , 2017, Physical review letters.

[114]  Allen Newell,et al.  Computer science as empirical inquiry: symbols and search , 1976, CACM.

[115]  Christos H. Papadimitriou,et al.  α-Rank: Multi-Agent Evaluation by Evolution , 2019, Scientific Reports.

[116]  David Silver,et al.  A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning , 2017, NIPS.

[117]  Jakub W. Pachocki,et al.  Dota 2 with Large Scale Deep Reinforcement Learning , 2019, ArXiv.

[118]  Michael William Newman,et al.  The Laplacian spectrum of graphs , 2001 .

[119]  Bryan Bruns,et al.  Escaping Prisoner's Dilemmas: From Discord to Harmony in the Landscape of 2x2 Games , 2012, ArXiv.

[120]  Peter Stone,et al.  Balancing Individual Preferences and Shared Objectives in Multiagent Reinforcement Learning , 2020, International Joint Conference on Artificial Intelligence.

[121]  William H. Sandholm,et al.  Population Games And Evolutionary Dynamics , 2010, Economic learning and social evolution.

[122]  Daphna Weinshall,et al.  On The Power of Curriculum Learning in Training Deep Networks , 2019, ICML.

[123]  Drew Fudenberg,et al.  Evolutionary game dynamics in finite populations , 2004, Bulletin of mathematical biology.

[124]  Danai Koutra,et al.  DELTACON: A Principled Massive-Graph Similarity Function , 2013, SDM.

[125]  Julian Togelius,et al.  Artificial Intelligence and Games , 2018, Springer International Publishing.

[126]  Julian Togelius,et al.  Rules and Mechanics , 2016 .

[127]  John H. Miller,et al.  Complex adaptive systems - an introduction to computational models of social life , 2009, Princeton studies in complexity.

[128]  Arthur L. Samuel,et al.  Programming Computers to Play Games , 1960, Adv. Comput..

[129]  Rui Wang,et al.  Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions , 2019, ArXiv.

[130]  S. Barry Cooper,et al.  Digital Computers Applied to Games , 2013 .

[131]  Michael R. Genesereth,et al.  General Game Playing: Overview of the AAAI Competition , 2005, AI Mag..

[132]  Frédéric Maire,et al.  Evolutionary Game Design , 2011, IEEE Transactions on Computational Intelligence and AI in Games.

[133]  D. Fudenberg,et al.  Emergence of cooperation and evolutionary stability in finite populations , 2004, Nature.

[134]  Branislav Bosanský,et al.  Combining Compact Representation and Incremental Generation in Large Games with Sequential Strategies , 2015, AAAI.

[135]  A. Rubinstein,et al.  The 11-20 Money Request Game: A Level-k Reasoning Study , 2012 .

[136]  Krzysztof R. Apt,et al.  A classification of weakly acyclic games , 2015 .

[137]  I. D. Hill,et al.  Faster than Thought. A Symposium on Digital Computing Machines , 1972 .

[138]  Alvaro Francisco Huertas-Rosero.  A Cartography for 2x2 Symmetric Games , 2003, ArXiv.

[139]  Avrim Blum,et al.  Planning in the Presence of Cost Functions Controlled by an Adversary , 2003, ICML.

[140]  D. Robinson,et al.  The topology of the 2x2 games : a new periodic table , 2005 .

[141]  Joel Z. Leibo,et al.  Multi-agent Reinforcement Learning in Sequential Social Dilemmas , 2017, AAMAS.

[142]  Jitendra Malik,et al.  Normalized Cuts and Image Segmentation , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[143]  Rémi Coulom,et al.  Efficient Selectivity and Backup Operators in Monte-Carlo Tree Search , 2006, Computers and Games.

[144]  J. Elman.  Learning and development in neural networks: the importance of starting small , 1993, Cognition.

[145]  Berkeley,et al.  Critical transitions in nature and society , 2009, Choice Reviews Online.

[146]  Michael Mateas,et al.  Towards Automated Game Design , 2007, AI*IA.

[147]  Kai A. Krueger,et al.  Flexible shaping: How learning in small steps helps , 2009, Cognition.

[148]  Hao Wang,et al.  Game reward systems: Gaming experiences and social meanings , 2011, DiGRA Conference.

[149]  César A. Hidalgo,et al.  The Atlas of Economic Complexity: Mapping Paths to Prosperity , 2011 .

[150]  Julian Togelius,et al.  Towards generating arcade game rules with VGDL , 2015, 2015 IEEE Conference on Computational Intelligence and Games (CIG).

[151]  David Robinson,et al.  Topology of 2x2 Games , 2005 .

[152]  V. Latora,et al.  Complex networks: Structure and dynamics , 2006 .

[153]  Stephen J. Roberts,et al.  Optimising Worlds to Evaluate and Influence Reinforcement Learning Agents , 2019, AAMAS.

[154]  N. Lazzaro.  Why We Play : Affect and the Fun of Games—Designing Emotions for Games, Entertainment Interfaces, and Interactive Products , 2012 .

[155]  M. Olson,et al.  The Logic of Collective Action , 1965 .

[156]  Karl Tuyls,et al.  Evolutionary Dynamics of Multi-Agent Learning: A Survey , 2015, J. Artif. Intell. Res..

[157]  King-Sun Fu,et al.  A distance measure between attributed relational graphs for pattern recognition , 1983, IEEE Transactions on Systems, Man, and Cybernetics.

[158]  Rajeev Motwani,et al.  The PageRank Citation Ranking : Bringing Order to the Web , 1999, WWW 1999.

[159]  José Hernández-Orallo,et al.  A New AI Evaluation Cosmos: Ready to Play the Game? , 2017, AI Mag..

[160]  Julian Togelius,et al.  Obstacle Tower: A Generalization Challenge in Vision, Control, and Planning , 2019, IJCAI.

[161]  Michael P. Wellman,et al.  Iterated Deep Reinforcement Learning in Games: History-Aware Training for Improved Stability , 2019, EC.

[162]  Taehoon Kim,et al.  Quantifying Generalization in Reinforcement Learning , 2018, ICML.

[163]  Thore Graepel,et al.  Re-evaluating evaluation , 2018, NeurIPS.

[164]  Thomas Wilhelm,et al.  What is a complex graph , 2008 .

[165]  D. Weinshall,et al.  Curriculum Learning by Transfer Learning: Theory and Experiments with Deep Networks , 2018, ICML.

[166]  Marc G. Bellemare,et al.  The Arcade Learning Environment: An Evaluation Platform for General Agents , 2012, J. Artif. Intell. Res..

[167]  M. Vitevitch.  What can graph theory tell us about word learning and lexical retrieval? , 2008, Journal of speech, language, and hearing research : JSLHR.

[168]  Demis Hassabis,et al.  A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play , 2018, Science.

[169]  Francisco C. Santos,et al.  Social norm complexity and past reputations in the evolution of cooperation , 2018, Nature.

[170]  Michael P. Wellman,et al.  Structure Learning for Approximate Solution of Many-Player Games , 2020, AAAI.

[171]  Julian Togelius,et al.  Procedural Content Generation in Games , 2016, Computational Synthesis and Creative Systems.

[172]  F. C. Santos,et al.  Emergence of fairness in repeated group interactions. , 2012, Physical review letters.

[173]  Herbert A. Simon,et al.  Computer Science as Empirical Inquiry , 2011 .

[174]  Constantinos Daskalakis,et al.  On the complexity of approximating a Nash equilibrium , 2011, SODA '11.