Newton-type Methods for Minimax Optimization

Differential games, in particular two-player sequential games (a.k.a. minimax optimization), have been an important modelling tool in applied science and received renewed interest in machine learning due to many recent applications. To account for the sequential and nonconvex nature, new solution concepts and algorithms have been developed. In this work, we provide a detailed analysis of existing algorithms and relate them to two novel Newton-type algorithms. We argue that our Newton-type algorithms nicely complement existing ones in that (a) they converge faster to (strict) local minimax points; (b) they are much more effective when the problem is ill-conditioned; (c) their computational complexity remains similar. We verify our theoretical results by conducting experiments on training GANs.

[1]  T. Rutherford,et al.  Nonlinear Programming , 2021, Mathematical Programming Methods for Geographers and Planners.

[2]  Tanner Fiez,et al.  Implicit Learning Dynamics in Stackelberg Games: Equilibria Characterization, Convergence Analysis, and Empirical Study , 2020, ICML.

[3]  Yaoliang Yu,et al.  Optimality and Stability in Non-Convex Smooth Games , 2020, J. Mach. Learn. Res..

[4]  Yaoliang Yu,et al.  Optimality and Stability in Non-Convex-Non-Concave Min-Max Optimization , 2020, ArXiv.

[5]  Asuman Ozdaglar,et al.  GANs May Have No Nash Equilibria , 2020, ICML 2020.

[6]  Junyu Zhang,et al.  On Lower Iteration Complexity Bounds for the Saddle Point Problems , 2019 .

[7]  Xingshi He,et al.  Introduction to Optimization , 2015, Applied Evolutionary Algorithms for Engineers Using Python.

[8]  Jimmy Ba,et al.  On Solving Minimax Optimization Locally: A Follow-the-Ridge Approach , 2019, ICLR.

[9]  J. Malick,et al.  On the convergence of single-call stochastic extra-gradient methods , 2019, NeurIPS.

[10]  Yaoliang Yu,et al.  Convergence of Gradient Methods on Bilinear Zero-Sum Games , 2019, ICLR.

[11]  Ioannis Mitliagkas,et al.  Linear Lower Bounds and Conditioning of Differentiable Games , 2019, ICML.

[12]  Ioannis Mitliagkas,et al.  Lower Bounds and Conditioning of Differentiable Games , 2019, ArXiv.

[13]  Lillian J. Ratliff,et al.  Convergence of Learning Dynamics in Stackelberg Games , 2019, ArXiv.

[14]  Michael I. Jordan,et al.  What is Local Optimality in Nonconvex-Nonconcave Minimax Optimization? , 2019, ICML.

[15]  Michael I. Jordan,et al.  Minmax Optimization: Stable Limit Points of Gradient Descent Ascent are Locally Optimal , 2019, ArXiv.

[16]  Mehryar Mohri,et al.  Agnostic Federated Learning , 2019, ICML.

[17]  Stefano Ermon,et al.  Learning Controllable Fair Representations , 2018, AISTATS.

[18]  Yangyang Xu,et al.  Lower complexity bounds of first-order methods for convex-concave bilinear saddle-point problems , 2018, Math. Program..

[19]  Chuan-Sheng Foo,et al.  Optimistic mirror descent in saddle-point problems: Going the extra (gradient) mile , 2018, ICLR.

[20]  Stephen J. Wright,et al.  A Newton-CG algorithm with complexity guarantees for smooth unconstrained optimization , 2018, Mathematical Programming.

[21]  Thore Graepel,et al.  The Mechanics of n-Player Differentiable Games , 2018, ICML.

[22]  Yuichi Yoshida,et al.  Spectral Normalization for Generative Adversarial Networks , 2018, ICLR.

[23]  Sashank J. Reddi,et al.  On the Convergence of Adam and Beyond , 2018, ICLR.

[24]  Le Song,et al.  SBEED: Convergent Reinforcement Learning with Nonlinear Function Approximation , 2017, ICML.

[25]  Shimon Whiteson,et al.  Learning with Opponent-Learning Awareness , 2017, AAMAS.

[26]  Léon Bottou,et al.  Wasserstein Generative Adversarial Networks , 2017, ICML.

[27]  Sepp Hochreiter,et al.  GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium , 2017, NIPS.

[28]  Aleksander Madry,et al.  Towards Deep Learning Models Resistant to Adversarial Attacks , 2017, ICLR.

[29]  J. Zico Kolter,et al.  Gradient descent GAN optimization is locally stable , 2017, NIPS.

[30]  Sebastian Nowozin,et al.  The Numerics of GANs , 2017, NIPS.

[31]  Lihong Li,et al.  Stochastic Variance Reduction Methods for Policy Evaluation , 2017, ICML.

[32]  Yann LeCun,et al.  Singularity of the Hessian in Deep Learning , 2016, ArXiv.

[33]  David Pfau,et al.  Unrolled Generative Adversarial Networks , 2016, ICLR.

[34]  François Laviolette,et al.  Domain-Adversarial Training of Neural Networks , 2015, J. Mach. Learn. Res..

[35]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[36]  Yurii Nesterov,et al.  Introductory Lectures on Convex Optimization - A Basic Course , 2014, Applied Optimization.

[37]  James Martens,et al.  Deep learning via Hessian-free optimization , 2010, ICML.

[38]  Stephen P. Boyd,et al.  Convex Optimization , 2004, IEEE Transactions on Automatic Control.

[39]  V. Borkar.  Stochastic Approximation: A Dynamical Systems Viewpoint , 2008 .

[40]  Robert Leonard.  Theory of Games and Economic Behavior , 2006 .

[41]  Carl D. Meyer,et al.  Matrix Analysis and Applied Linear Algebra , 2000 .

[42]  Barak A. Pearlmutter.  Fast Exact Multiplication by the Hessian , 1994, Neural Computation.

[43]  P. Werbos.  Backpropagation: past and future , 1988, ICNN.

[44]  L. Popov.  A modification of the Arrow-Hurwicz method for search of saddle points , 1980 .

[45]  J. Nocedal.  Updating Quasi-Newton Matrices With Limited Storage , 1980 .

[46]  B. V. Dean,et al.  Studies in Linear and Non-Linear Programming. , 1959 .

[47]  J. Neumann,et al.  Theory of games and economic behavior , 1945, 100 Years of Math Milestones.

[48]  Asuman Ozdaglar,et al.  Do GANs always have Nash equilibria? , 2020, ICML.

[49]  Jinsung Yoon,et al.  GENERATIVE ADVERSARIAL NETS , 2018 .

[50]  John C. Duchi,et al.  Certifiable Distributional Robustness with Principled Adversarial Training , 2017, ArXiv.

[51]  Geoffrey E. Hinton,et al.  Supporting Online Material for Reducing the Dimensionality of Data with Neural Networks , 2006 .

[52]  Arkadi Nemirovski,et al.  Prox-Method with Rate of Convergence O(1/t) for Variational Inequalities with Lipschitz Continuous Monotone Operators and Smooth Convex-Concave Saddle Point Problems , 2004, SIAM J. Optim..

[53]  E. Chong,et al.  Introduction to optimization , 1987 .

[54]  G. M. Korpelevich.  The extragradient method for finding saddle points and other problems , 1976 .

[55]  Yu. G. Evtushenko,et al.  Some local properties of minimax problems , 1974 .

[56]  Yu. G. Evtushenko,et al.  Iterative methods for solving minimax problems , 1974 .

[57]  Boris Polyak.  Some methods of speeding up the convergence of iteration methods , 1964 .

[58]  J. Nash.  Equilibrium Points in N-Person Games. , 1950, Proceedings of the National Academy of Sciences of the United States of America.