Empirical Game-Theoretic Analysis in Mean Field Games

We present a simulation-based approach for solution of mean field games (MFGs), using the framework of empirical game-theoretical analysis (EGTA). Our method employs a version of the double oracle, iteratively adding strategies based on best response to the equilibrium of the empirical MFG among strategies considered so far. We present Fictitious Play (FP) and Replicator Dynamics as two subroutines for computing the empirical game equilibrium. Each subroutine is implemented with a query-based method rather than maintaining an explicit payoff matrix as in typical EGTA methods due to a representation issue we highlight for MFGs. We prove that a Nash equilibrium (NE) exists in the empirical MFG and show the convergence of iterative EGTA to NE of the full MFG with either subroutine. We test the performance of iterative EGTA in various games and show that it outperforms directly applying FP to MFGs in terms of iterations of strategy introduction.

[1]  Csaba Szepesvári,et al.  Bounds and dynamics for empirical game theoretic analysis , 2019, Autonomous Agents and Multi-Agent Systems.

[2]  Matthieu Geist,et al.  Scaling up Mean Field Games with Online Mirror Descent , 2021, ArXiv.

[3]  B. Gaujal,et al.  Discrete mean field games: Existence of equilibria and convergence , 2019, Journal of Dynamics & Games.

[4]  Sriram Srinivasan,et al.  OpenSpiel: A Framework for Reinforcement Learning in Games , 2019, ArXiv.

[5]  Max Jaderberg,et al.  Real World Games Look Like Spinning Tops , 2020, NeurIPS.

[6]  C. Berge.  Topological Spaces: including a treatment of multi-valued functions , 2010 .

[7]  Wojciech M. Czarnecki,et al.  Grandmaster level in StarCraft II using multi-agent reinforcement learning , 2019, Nature.

[8]  Michael P. Wellman,et al.  Empirical Mechanism Design for Optimizing Clearing Interval in Frequent Call Markets , 2017, EC.

[9]  Keith B. Hall,et al.  Fair and Efficient Solutions to the Santa Fe Bar Problem , 1910 .

[10]  Matthieu Geist,et al.  Generalization in Mean Field Games by Learning Master Policies , 2021, AAAI.

[11]  J. Robinson.  AN ITERATIVE METHOD OF SOLVING A GAME , 1951, Classics in Game Theory.

[12]  P. Lions,et al.  Mean field games , 2007 .

[13]  J. M. Smith,et al.  The Logic of Animal Conflict , 1973, Nature.

[14]  Romuald Elie,et al.  Fictitious Play for Mean Field Games: Continuous Time Analysis and Applications , 2020, NeurIPS.

[15]  Olivier Pietquin,et al.  Learning Equilibria in Mean-Field Games: Introducing Mean-Field PSRO , 2021, ArXiv.

[16]  P. Taylor,et al.  Evolutionarily Stable Strategies and Game Dynamics , 1978 .

[17]  Lantao Yu,et al.  Deep Reinforcement Learning for Green Security Games with Real-Time Information , 2018, AAAI.

[18]  W. Browder,et al.  Annals of Mathematics , 1889 .

[19]  Matthieu Geist,et al.  Mean Field Games Flock! The Reinforcement Learning Way , 2021, IJCAI.

[20]  On the Convergence of Model Free Learning in Mean Field Games , 2019, AAAI.

[21]  Peter E. Caines,et al.  Large population stochastic dynamic games: closed-loop McKean-Vlasov systems and the Nash certainty equivalence principle , 2006, Commun. Inf. Syst..

[22]  W. Arthur.  Inductive Reasoning and Bounded Rationality , 1994 .

[23]  M CarlsteadSara,et al.  The Grace Hopper Celebration of Women in Computing , 1994 .

[24]  Annales scientifiques de ľÉcole Normale Supérieure , 1891 .

[25]  Yoram Bachrach,et al.  Sample-based Approximation of Nash in Large Many-Player Games via Gradient Descent , 2021, Adaptive Agents and Multi-Agent Systems.

[26]  O. H. Brownlee,et al.  ACTIVITY ANALYSIS OF PRODUCTION AND ALLOCATION , 1952 .

[27]  David Silver,et al.  A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning , 2017, NIPS.

[28]  P. Cochat,et al.  Et al , 2008, Archives de pediatrie : organe officiel de la Societe francaise de pediatrie.

[29]  Michael P. Wellman,et al.  Methods for empirical game-theoretic analysis (extended abstract) , 2006 .

[30]  Avrim Blum,et al.  Planning in the Presence of Cost Functions Controlled by an Adversary , 2003, ICML.

[31]  Michael Wooldridge,et al.  Autonomous agents and multi-agent systems , 2014 .

[32]  L. Desaint,et al.  Sur quelques points de la théorie des fonctions , 1897 .

[33]  Mikio Kobayashi.  Japanese Journal of Mathematics , 2009 .

[34]  S. Kakutani.  A generalization of Brouwer’s fixed point theorem , 1941 .

[35]  Michael P. Wellman,et al.  Iterated Deep Reinforcement Learning in Games: History-Aware Training for Improved Stability , 2019, EC.

[36]  D. Gomes,et al.  Discrete Time, Finite State Space Mean Field Games , 2010 .