Empirical Game-Theoretic Analysis for Mean Field Games

We present a simulation-based approach for solution of mean field games (MFGs), using the framework of empirical game-theoretical analysis (EGTA). Our primary method employs a version of the double oracle, iteratively adding strategies based on best response to the equilibrium of the empirical MFG among strategies considered so far. We present Fictitious Play (FP) and Replicator Dynamics as two subroutines for computing the empirical game equilibrium. Each subroutine is implemented with a query-based method rather than maintaining an explicit payoff matrix as in typical EGTA methods due to a representation issue we highlight for MFGs. By introducing game model learning and regularization, we significantly improve the sample efficiency of the primary method without sacrificing the overall learning performance. Theoretically, we prove that a Nash equilibrium (NE) exists in the empirical MFG and show the convergence of iterative EGTA to NE of the full MFG with either subroutine. We test the performance of iterative EGTA in various games and show that it outperforms directly applying FP to MFGs in terms of iterations of strategy introduction.

[1]  Michael P. Wellman,et al.  Regularization for Strategy Exploration in Empirical Game-Theoretic Analysis , 2023, ArXiv.

[2]  Mark Rowland,et al.  Learning Equilibria in Mean-Field Games: Introducing Mean-Field PSRO , 2021, AAMAS.

[3]  Matthieu Geist,et al.  Generalization in Mean Field Games by Learning Master Policies , 2021, AAAI.

[4]  Yoram Bachrach,et al.  Sample-based Approximation of Nash in Large Many-Player Games via Gradient Descent , 2021, Adaptive Agents and Multi-Agent Systems.

[5]  Matthieu Geist,et al.  Mean Field Games Flock! The Reinforcement Learning Way , 2021, IJCAI.

[6]  Matthieu Geist,et al.  Scaling up Mean Field Games with Online Mirror Descent , 2021, ArXiv.

[7]  Romuald Elie,et al.  Fictitious Play for Mean Field Games: Continuous Time Analysis and Applications , 2020, NeurIPS.

[8]  Max Jaderberg,et al.  Real World Games Look Like Spinning Tops , 2020, NeurIPS.

[9]  O. Pietquin,et al.  On the Convergence of Model Free Learning in Mean Field Games , 2019, AAAI.

[10]  J. Robinson.  AN ITERATIVE METHOD OF SOLVING A GAME , 1951, Classics in Game Theory.

[11]  Csaba Szepesvári,et al.  Bounds and dynamics for empirical game theoretic analysis , 2019, Autonomous Agents and Multi-Agent Systems.

[12]  Wojciech M. Czarnecki,et al.  Grandmaster level in StarCraft II using multi-agent reinforcement learning , 2019, Nature.

[13]  Sriram Srinivasan,et al.  OpenSpiel: A Framework for Reinforcement Learning in Games , 2019, ArXiv.

[14]  Samuel Sokota,et al.  Learning Deviation Payoffs in Simulation-Based Games , 2019, AAAI.

[15]  B. Gaujal,et al.  Discrete mean field games: Existence of equilibria and convergence , 2019, Journal of Dynamics & Games.

[16]  Michael P. Wellman,et al.  Iterated Deep Reinforcement Learning in Games: History-Aware Training for Improved Stability , 2019, EC.

[17]  Lantao Yu,et al.  Deep Reinforcement Learning for Green Security Games with Real-Time Information , 2018, AAAI.

[18]  Michael P. Wellman,et al.  A Regression Approach for Modeling Games With Many Symmetric Players , 2018, AAAI.

[19]  David Silver,et al.  A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning , 2017, NIPS.

[20]  Michael P. Wellman,et al.  Empirical Mechanism Design for Optimizing Clearing Interval in Frequent Call Markets , 2017, EC.

[21]  P. Cochat,et al.  Et al , 2008, Archives de pediatrie : organe officiel de la Societe francaise de pediatrie.

[22]  D. Gomes,et al.  Discrete Time, Finite State Space Mean Field Games , 2010 .

[23]  Mikio Kobayashi.  Japanese Journal of Mathematics , 2009 .

[24]  David C. Parkes,et al.  Learning and Solving Many-Player Games through a Cluster-Based Representation , 2008, UAI.

[25]  P. Lions,et al.  Mean field games , 2007 .

[26]  Michael P. Wellman,et al.  Learning payoff functions in infinite games , 2005, Machine Learning.

[27]  Peter E. Caines,et al.  Large population stochastic dynamic games: closed-loop McKean-Vlasov systems and the Nash certainty equivalence principle , 2006, Commun. Inf. Syst..

[28]  Michael P. Wellman,et al.  Methods for empirical game-theoretic analysis (extended abstract) , 2006 .

[29]  Avrim Blum,et al.  Planning in the Presence of Cost Functions Controlled by an Adversary , 2003, ICML.

[30]  Michael Wooldridge,et al.  Autonomous agents and multi-agent systems , 2014 .

[31]  M CarlsteadSara,et al.  The Grace Hopper Celebration of Women in Computing , 1994 .

[32]  W. Arthur.  Inductive Reasoning and Bounded Rationality , 1994 .

[33]  C. Berge.  Topological Spaces: including a treatment of multi-valued functions , 2010 .

[34]  P. Taylor,et al.  Evolutionarily Stable Strategies and Game Dynamics , 1978 .

[35]  J. M. Smith,et al.  The Logic of Animal Conflict , 1973, Nature.

[36]  O. H. Brownlee,et al.  ACTIVITY ANALYSIS OF PRODUCTION AND ALLOCATION , 1952 .

[37]  S. Kakutani.  A generalization of Brouwer’s fixed point theorem , 1941 .

[38]  Keith B. Hall,et al.  Fair and Efficient Solutions to the Santa Fe Bar Problem , 1910 .

[39]  L. Desaint,et al.  Sur quelques points de la théorie des fonctions , 1897 .

[40]  Numérisation de documents anciens mathématiques,et al.  Annales scientifiques de ľÉcole Normale Supérieure , 1893 .

[41]  W. Browder,et al.  Annals of Mathematics , 1889 .