A policy iteration method for mean field games

The policy iteration method is a classical algorithm for solving optimal control problems. In this paper, we introduce a policy iteration method for Mean Field Games systems, and we study the convergence of this procedure to a solution of the problem. We also introduce suitable discretizations to numerically solve both stationary and evolutive problems. We show the convergence of the policy iteration method for the discrete problem and we study the performance of the proposed algorithm on some examples in dimension one and two.

[1]  W. D. Evans,et al.  PARTIAL DIFFERENTIAL EQUATIONS , 1941 .

[2]  Ronald A. Howard,et al.  Dynamic Programming and Markov Processes , 1960 .

[3]  R. Bellman.  Dynamic programming. , 1957, Science.

[4]  O. Ladyženskaja.  Linear and Quasilinear Equations of Parabolic Type , 1968 .

[5]  M. Puterman.  Optimal control of diffusion processes with reflection , 1977 .

[6]  Martin L. Puterman,et al.  On the Convergence of Policy Iteration in Stationary Dynamic Programming , 1979, Math. Oper. Res..

[7]  Martin L. Puterman,et al.  On the convergence of policy iteration for controlled diffusions , 1981 .

[8]  P. Lions,et al.  Quelques remarques sur les problemes elliptiques quasilineaires du second ordre , 1985 .

[9]  H. Triebel,et al.  Topics in Fourier Analysis and Function Spaces , 1987 .

[10]  O. A. Ladyzhenskai︠a︡,et al.  Linear and Quasi-linear Equations of Parabolic Type , 1995 .

[11]  Roland Schnaubelt,et al.  Solvability and Maximal Regularity of Parabolic Evolution Equations with Coefficients Continuous in Time , 2001 .

[12]  John Rust,et al.  Convergence Properties of Policy Iteration , 2003, SIAM J. Control. Optim..

[13]  Philippe Souplet,et al.  Global solutions of inhomogeneous Hamilton-Jacobi equations , 2006 .

[14]  P. Lions,et al.  Mean field games , 2007 .

[15]  Minyi Huang,et al.  Large-Population Cost-Coupled LQG Problems With Nonuniform Agents: Individual-Mass Behavior and Decentralized $\varepsilon$-Nash Equilibria , 2007, IEEE Transactions on Automatic Control.

[16]  A. Rhandi,et al.  Global properties of transition pProbabilities of singular diffusions@@@Global properties of transition pProbabilities of singular diffusions , 2009 .

[17]  Hasnaa Zidani,et al.  Some Convergence Results for Howard's Algorithm , 2009, SIAM J. Numer. Anal..

[18]  Abdelaziz Rhandi,et al.  GLOBAL PROPERTIES OF TRANSITION PROBABILITIES OF SINGULAR DIFFUSIONS , 2010 .

[19]  Lawrence C. Evans,et al.  Adjoint and Compensated Compactness Methods for Hamilton–Jacobi PDE , 2010 .

[20]  Yves Achdou,et al.  Mean Field Games: Numerical Methods , 2010, SIAM J. Numer. Anal..

[21]  Pierre-Louis Lions,et al.  Long time average of mean field games , 2012, Networks Heterog. Media.

[22]  Yves Achdou,et al.  Mean Field Games: Convergence of a Finite Difference Method , 2012, SIAM J. Numer. Anal..

[23]  Elisabetta Carlini,et al.  A Semi-Lagrangian scheme for a degenerate second order Mean Field Game system , 2014, 1404.5932.

[24]  Fabio S. Priuli,et al.  Linear-Quadratic N-person and Mean-Field Games with Ergodic Cost , 2014, SIAM J. Control. Optim..

[25]  P. Cardaliaguet,et al.  Second order mean field games with degenerate diffusion and local coupling , 2014, 1407.7024.

[26]  Alessandro Alla,et al.  An Efficient Policy Iteration Algorithm for Dynamic Programming Equations , 2013, SIAM J. Sci. Comput..

[27]  Vladimir I. Bogachev,et al.  Fokker-planck-kolmogorov Equations , 2015 .

[28]  Y. Achdou,et al.  ON THE SYSTEM OF PARTIAL DIFFERENTIAL EQUATIONS ARISING IN MEAN FIELD TYPE CONTROL , 2015, 1503.05044.

[29]  Pierre Cardaliaguet,et al.  Learning in mean field games: The fictitious play , 2015, 1507.06280.

[30]  Martino Bardi,et al.  Nonlinear elliptic systems and mean-field games , 2016 .

[31]  Rene Carmona,et al.  Mean Field Games of Timing and Models for Bank Runs , 2016, 1606.03709.

[32]  Simone Cacace,et al.  A Generalized Newton Method for Homogenization of Hamilton-Jacobi Equations , 2016, SIAM J. Sci. Comput..

[33]  P. Cardaliaguet,et al.  Stable solutions in potential mean field game systems , 2016, 1612.01877.

[34]  Luigi Orsina,et al.  Strongly coupled elliptic equations related to mean-field games systems , 2016 .

[35]  Maria Colombo,et al.  Optimality of integrability estimates for advection–diffusion equations , 2017, 1702.00321.

[36]  Alessio Porretta,et al.  On the weak theory for mean field games systems , 2017 .

[37]  A. Porretta,et al.  On the turnpike property for mean field games , 2018 .

[38]  Dante Kalise,et al.  Proximal Methods for Stationary Mean Field Games with Local Couplings , 2016, SIAM J. Control. Optim..

[39]  Marco Cirant,et al.  On the Existence and Uniqueness of Solutions to Time-Dependent Fractional MFG , 2018, SIAM J. Math. Anal..

[40]  Mathieu Lauriere,et al.  Mean Field Games and Applications: Numerical Aspects , 2020, Lecture Notes in Mathematics.

[41]  Marco Cirant,et al.  On the Problem of Maximal $$L^q$$-regularity for Viscous Hamilton–Jacobi Equations , 2020, Archive for Rational Mechanics and Analysis.

[42]  Marco Cirant,et al.  Lipschitz regularity for viscous Hamilton-Jacobi equations with L terms , 2018, Annales de l'Institut Henri Poincaré C, Analyse non linéaire.

[43]  Roberto Gianni,et al.  Short-Time Existence for a General Backward–Forward Parabolic System Arising from Mean-Field Games , 2018, Dyn. Games Appl..

[44]  L. Szpruch,et al.  Exponential Convergence and stability of Howards's Policy Improvement Algorithm for Controlled Diffusions , 2018, 1812.07846.