Bounds and dynamics for empirical game theoretic analysis

This paper provides several theoretical results for empirical game theory. Specifically, we introduce bounds for empirical game theoretical analysis of complex multi-agent interactions. In doing so we provide insights in the empirical meta game showing that a Nash equilibrium of the estimated meta-game is an approximate Nash equilibrium of the true underlying meta-game. We investigate and show how many data samples are required to obtain a close enough approximation of the underlying game. Additionally, we extend the evolutionary dynamics analysis of meta-games using heuristic payoff tables (HPTs) to asymmetric games. The state-of-the-art has only considered evolutionary dynamics of symmetric HPTs in which agents have access to the same strategy sets and the payoff structure is symmetric, implying that agents are interchangeable. Finally, we carry out an empirical illustration of the generalised method in several domains, illustrating the theory and evolutionary dynamics of several versions of the AlphaGo algorithm (symmetric), the dynamics of the Colonel Blotto game played by human players on Facebook (symmetric), the dynamics of several teams of players in the capture the flag game (symmetric), and an example of a meta-game in Leduc Poker (asymmetric), generated by the policy-space response oracle multi-agent learning algorithm.

[1]  É. Borel.  The Theory of Play and Integral Equations with Skew Symmetric Kernels , 1953 .

[2]  J. M. Smith,et al.  The Logic of Animal Conflict , 1973, Nature.

[3]  E. C. Zeeman,et al.  Population dynamics from game theory , 1980 .

[4]  E. Zeeman.  Dynamics of the evolution of animal conflicts , 1981 .

[5]  Jörgen W. Weibull,et al.  Evolutionary Game Theory , 1996 .

[6]  Josef Hofbauer,et al.  Evolutionary Games and Population Dynamics , 1998 .

[7]  Manuela M. Veloso,et al.  Towards collaborative and adversarial learning: a case study in robotic soccer , 1998, Int. J. Hum. Comput. Stud..

[8]  Herbert Gintis,et al.  Game Theory Evolving: A Problem-Centered Introduction to Modeling Strategic Interaction - Second Edition , 2009 .

[9]  G. Tesauro,et al.  Analyzing Complex Strategic Interactions in Multi-Agent Systems , 2002 .

[10]  Tom Lenaerts,et al.  A selection-mutation model for q-learning in multi-agent systems , 2003, AAMAS '03.

[11]  Rajarshi Das,et al.  Choosing Samples to Compute Heuristic-Strategy Nash Equilibrium , 2003, AMEC.

[12]  Peter McBurney,et al.  An evolutionary game-theoretic comparison of two double-auction market designs , 2004, AAMAS'04.

[13]  Michael H. Bowling,et al.  Bayes' Bluff: Opponent Modelling in Poker , 2005, UAI 2005.

[14]  Peyman Faratin,et al.  Agent-Mediated Electronic Commerce VI, Theories for and Engineering of Distributed Mechanisms and Systems, AAMAS 2004 Workshop, AMEC 2004, New York, NY, USA, July 19, 2004, Revised Selected Papers , 2005, AMEC.

[15]  Michael P. Wellman.  Methods for Empirical Game-Theoretic Analysis , 2006, AAAI.

[16]  Elizabeth Sklar,et al.  Auctions, Evolution, and Multi-agent Learning , 2007, Adaptive Agents and Multi-Agents Systems.

[17]  Alvin E. Roth,et al.  Multi-agent learning and the descriptive value of simple models , 2007, Artif. Intell..

[18]  Michael P. Wellman,et al.  Learning payoff functions in infinite games , 2005, Machine Learning.

[19]  Simon Parsons,et al.  What evolutionary game theory tells us about multiagent learning , 2007, Artif. Intell..

[20]  Rémi Coulom,et al.  Whole-History Rating: A Bayesian Rating System for Players of Time-Varying Strength , 2008, Computers and Games.

[21]  Simon Parsons,et al.  Auction Analysis by Normal Form Game Approximation , 2008, 2008 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology.

[22]  Michael P. Wellman,et al.  Stochastic Search Methods for Nash Equilibrium Approximation in Simulation-based Games , 2022 .

[23]  Daniel Kudenko,et al.  Adaptive Agents and Multi-Agent Systems III. Adaptation and Multi-Agent Learning, 5th, 6th, and 7th European Symposium, ALAMAS 2005-2007 on Adaptive and Learning Agents and Multi-Agent Systems, Revised Selected Papers , 2008, Adaptive Agents and Multi-Agent Systems.

[24]  Michael P. Wellman,et al.  Stronger CDA strategies through empirical game-theoretic analysis and reinforcement learning , 2009, AAMAS.

[25]  Jan Ramon,et al.  An evolutionary game-theoretic analysis of poker strategies , 2009, Entertain. Comput..

[26]  D. Avis,et al.  Enumeration of Nash equilibria for two-player games , 2010 .

[27]  Yevgeniy Vorobeychik,et al.  Probabilistic analysis of simulation-based games , 2010, TOMC.

[28]  Pushmeet Kohli,et al.  Colonel Blotto on Facebook: the effect of social relations on strategic interaction , 2012, WebSci '12.

[29]  Michael P. Wellman,et al.  EGTAOnline: An Experiment Manager for Simulation-Based Game Studies , 2012, MABS.

[30]  Michael P. Wellman,et al.  Scaling simulation-based game analysis through deviation-preserving reduction , 2012, AAMAS.

[31]  Mark H. M. Winands,et al.  Monte Carlo Tree Search for the Hide-and-Seek Game Scotland Yard , 2012, IEEE Transactions on Computational Intelligence and AI in Games.

[32]  Michael P. Wellman,et al.  Analyzing Incentives for Protocol Compliance in Complex Domains: A Case Study of Introduction-Based Routing , 2013, ArXiv.

[33]  Michael P. Wellman,et al.  Bootstrap statistics for empirical games , 2014, AAMAS.

[34]  Michael P. Wellman,et al.  Empirical Game-Theoretic Analysis for Moving Target Defense , 2015, MTD@CCS.

[35]  Michael P. Wellman,et al.  Welfare Effects of Market Making in Continuous Double Auctions , 2015, AAMAS.

[36]  Karl Tuyls,et al.  Evolutionary Dynamics of Multi-Agent Learning: A Survey , 2015, J. Artif. Intell. Res..

[37]  Peter McBurney,et al.  Trading in markets with noisy information: an evolutionary analysis , 2015, Connect. Sci..

[38]  Michael P. Wellman,et al.  Strategic Market Choice: Frequent Call Markets vs. Continuous Double Auctions for Fast and Slow Traders , 2015, EAI Endorsed Trans. Serious Games.

[39]  Mason Wright,et al.  Using Reinforcement Learning to Validate Empirical Game-Theoretic Analysis: A Continuous Double Auction Study , 2016, ArXiv.

[40]  Demis Hassabis,et al.  Mastering the game of Go with deep neural networks and tree search , 2016, Nature.

[41]  Michael P. Wellman,et al.  Moving Target Defense against DDoS Attacks: An Empirical Game-Theoretic Analysis , 2016, MTD@CCS.

[42]  D. M. V. Hesteren.  Evolutionary Game Theory , 2017 .

[43]  David Silver,et al.  A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning , 2017, NIPS.

[44]  Michael P. Wellman,et al.  Multi-Stage Attack Graph Security Games: Heuristic Strategies, with Empirical Game-Theoretic Analysis , 2017, MTD@CCS.

[45]  Michael P. Wellman,et al.  Evaluating the Stability of Non-Adaptive Trading in Continuous Double Auctions: A Reinforcement Learning Approach , 2018, AAAI Workshops.

[46]  Michael P. Wellman,et al.  A Cloaking Mechanism to Mitigate Market Manipulation , 2018, IJCAI.

[47]  Sarit Kraus,et al.  Predicting Human Decision-Making: From Prediction to Action , 2018, Predicting Human Decision-Making.

[48]  Joel Z. Leibo,et al.  A Generalised Method for Empirical Game Theoretic Analysis , 2018, AAMAS.

[49]  Shane Legg,et al.  Symmetric Decomposition of Asymmetric Games , 2017, Scientific Reports.

[50]  Joel Z. Leibo,et al.  Human-level performance in first-person multiplayer games with population-based deep reinforcement learning , 2018, ArXiv.

[51]  Michael P. Wellman,et al.  Multistage Attack Graph Security Games: Heuristic Strategies, with Empirical Game-Theoretic Analysis , 2018, Secur. Commun. Networks.

[52]  Arunesh Sinha,et al.  Bounding Regret in Simulated Games , 2018 .