Stability of Gradient Learning Dynamics in Continuous Games: Vector Action Spaces

Towards characterizing the optimization landscape of games, this paper analyzes the stability and spectrum of gradient-based dynamics near fixed points of two-player continuous games. We introduce the quadratic numerical range as a method to bound the spectrum of game dynamics linearized about local equilibria. We also analyze the stability of differential Nash equilibria and their robustness to variation in agent’s learning rates. Our results show that by decomposing the game Jacobian into symmetric and anti-symmetric components, we can assess the contribution of vector field’s potential and rotational components to the stability of the equilibrium. In zero-sum games, all differential Nash equilibria are stable; in potential games, all stable points are Nash. Furthermore, zero-sum Nash equilibria are robust in the sense that they are stable for all learning rates. For continuous games with general costs, we provide a sufficient condition for instability. We conclude with a numerical example that investigates how players with different learning rates can take advantage of rotational components of the game to converge faster.

[1]  Maryam Kamgarpour,et al.  Learning Nash Equilibria in Monotone Games , 2019, 2019 IEEE 58th Conference on Decision and Control (CDC).

[2]  Francis J. Doyle,et al.  Nonlinear systems theory , 1997 .

[3]  S. Shankar Sastry,et al.  On Gradient-Based Learning in Continuous Games , 2018, SIAM J. Math. Data Sci..

[4]  Tanner Fiez,et al.  Implicit Learning Dynamics in Stackelberg Games: Equilibria Characterization, Convergence Analysis, and Empirical Study , 2020, ICML.

[5]  S. Shankar Sastry,et al.  Genericity and structural stability of non-degenerate differential Nash equilibria , 2014, 2014 American Control Conference.

[6]  Georgios Piliouras,et al.  From Darwin to Poincaré and von Neumann: Recurrence and Cycles in Evolutionary and Algorithmic Game Theory , 2019, WINE.

[7]  Christos H. Papadimitriou,et al.  Cycles in adversarial regularized learning , 2017, SODA.

[8]  David Pfau,et al.  Unrolled Generative Adversarial Networks , 2016, ICLR.

[9]  S. Shankar Sastry,et al.  On the Characterization of Local Nash Equilibria in Continuous Games , 2014, IEEE Transactions on Automatic Control.

[10]  Na Li,et al.  Distributed Zero-Order Algorithms for Nonconvex Multi-Agent optimization , 2019, 2019 57th Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[11]  S. Shankar Sastry,et al.  Characterization and computation of local Nash equilibria in continuous games , 2013, 2013 51st Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[12]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[13]  Lillian J. Ratliff,et al.  Stability of Gradient Learning Dynamics in Continuous Games: Scalar Action Spaces , 2020, 2020 59th IEEE Conference on Decision and Control (CDC).

[14]  Lillian J. Ratliff,et al.  Global Convergence of Policy Gradient for Sequential Zero-Sum Linear Quadratic Dynamic Games , 2019, ArXiv.

[15]  Eric van Damme,et al.  Non-Cooperative Games , 2000 .

[16]  S. Sastry.  Nonlinear Systems: Analysis, Stability, and Control , 1999 .

[17]  C. Tretter.  Spectral Theory Of Block Operator Matrices And Applications , 2008 .

[18]  Zhengyuan Zhou,et al.  Learning in games with continuous action sets and unknown payoff functions , 2019, Math. Program..

[19]  Joel Z. Leibo,et al.  Smooth markets: A basic mechanism for organizing gradient-based learners , 2020, ICLR.

[20]  Christiane Tretter,et al.  Spectral inclusion for unbounded block operator matrices , 2009 .

[21]  L. Shapley,et al.  Potential Games , 1994 .

[22]  Pascal Vincent,et al.  A Closer Look at the Optimization Landscapes of Generative Adversarial Networks , 2019, ICLR.

[23]  Lillian J. Ratliff,et al.  Convergence Analysis of Gradient-Based Learning in Continuous Games , 2019, UAI.

[24]  D. Fudenberg,et al.  The Theory of Learning in Games , 1998 .

[25]  Christiane Tretter,et al.  A new concept for block operator matrices:the quadratic numerical range , 2001 .