A Continuous-Time Perspective on Optimal Methods for Monotone Equation Problems

We study rescaled gradient dynamical systems in a Hilbert space H , where implicit discretization in a finite-dimensional Euclidean space leads to high-order methods for solving monotone equations (MEs). Our framework can be interpreted as a natural generalization of celebrated dual extrapolation method [Nesterov, 2007] from first order to high order via appeal to the regularization toolbox of optimization theory [Nesterov, 2021a,b]. More specifically, we establish the existence and uniqueness of a global solution and analyze the convergence properties of solution trajectories. We also present discrete-time counterparts of our high-order continuous-time methods, and we show that the p th order method achieves an ergodic rate of O ( k − ( p +1) / 2 ) in terms of a restricted merit function and a pointwise rate of O ( k − p/ 2 ) in terms of a residue function. Under regularity conditions, the restarted version of p th -order methods achieves local convergence with the order p ≥ 2. Notably, our methods are optimal since they have matched the lower bound established for solving the monotone equation problems under a standard linear span assumption [Lin and Jordan, 2022].

[1]  Aryan Mokhtari,et al.  Generalized Optimistic Methods for Convex-Concave Saddle Point Problems , 2022, ArXiv.

[2]  H. Attouch,et al.  First-order inertial algorithms involving dry friction damping , 2021, Mathematical programming.

[3]  Shuzhong Zhang,et al.  On lower iteration complexity bounds for the convex concave saddle point problems , 2019, Math. Program..

[4]  Zaki Chbani,et al.  First-order optimization algorithms via inertial systems with Hessian driven damping , 2019, Mathematical Programming.

[5]  J. Renegar,et al.  A Simple Nearly Optimal Restart Scheme For Speeding Up First-Order Methods , 2018, Foundations of Computational Mathematics.

[6]  Michael I. Jordan,et al.  Perseus: A Simple High-Order Regularization Method for Variational Inequalities , 2022, ArXiv.

[7]  On Monotone Inclusions, Acceleration and Closed-Loop Control , 2021, ArXiv.

[8]  H. Attouch,et al.  Asymptotic behavior of Newton-like inertial dynamics involving the sum of potential and nonpotential terms , 2021, Fixed Point Theory and Algorithms for Sciences and Engineering.

[9]  Paul-Emile Maingé,et al.  First-Order Frameworks for Continuous Newton-like Dynamics Governed by Maximally Monotone Operators , 2021, Set-Valued and Variational Analysis.

[10]  Variants of the A-HPE and large-step A-HPE algorithms for strongly convex problems with applications to accelerated high-order tensor methods , 2021, 2102.02045.

[11]  Chaobing Song,et al.  Unified Acceleration of High-Order Algorithms under General Hölder Continuity , 2021, SIAM J. Optim..

[12]  H. Attouch,et al.  Fast convex optimization via inertial dynamics combining viscous and Hessian-driven damping with time rescaling , 2020, Evolution Equations & Control Theory.

[13]  Michael I. Jordan,et al.  Optimization with Momentum: Dynamical, Control-Theoretic, and Symplectic Perspectives , 2020, J. Mach. Learn. Res..

[14]  Michael I. Jordan,et al.  A control-theoretic perspective on optimal high-order optimization , 2019, Mathematical Programming.

[15]  Yurii Nesterov,et al.  Implementable tensor methods in unconstrained convex optimization , 2019, Mathematical Programming.

[16]  Yurii Nesterov,et al.  On inexact solution of auxiliary problems in tensor methods for convex optimization , 2021, Optim. Methods Softw..

[17]  Michael I. Jordan,et al.  Generalized Momentum-Based Methods: A Hamiltonian Perspective , 2019, SIAM J. Optim..

[18]  Michael I. Jordan,et al.  Understanding the acceleration phenomenon via high-resolution differential equations , 2018, Mathematical Programming.

[19]  Yangyang Xu,et al.  Lower complexity bounds of first-order methods for convex-concave bilinear saddle-point problems , 2018, Math. Program..

[20]  Peng Xu,et al.  Inexact Non-Convex Newton-Type Methods , 2018, 1802.06925.

[21]  Michael I. Jordan,et al.  A Lyapunov Analysis of Accelerated Methods in Optimization , 2021, J. Mach. Learn. Res..

[22]  O. Shamir,et al.  High-Order Oracle Complexity of Smooth and Strongly Convex Optimization , 2020, ArXiv.

[23]  Samir Adly,et al.  Finite Convergence of Proximal-Gradient Inertial Algorithms Combining Dry Friction with Hessian-Driven Damping , 2020, SIAM J. Optim..

[24]  Mouhacine Benosman,et al.  Finite-Time Convergence in Continuous-Time Optimization , 2020, ICML.

[25]  Kevin A. Lai,et al.  Higher-order methods for convex-concave min-max optimization and monotone variational inequalities , 2020, SIAM J. Optim..

[26]  E. R. Csetnek Continuous Dynamics Related to Monotone Inclusions and Non-Smooth Optimization Problems , 2020, 2007.00460.

[27]  H. Attouch,et al.  Continuous Newton-like Inertial Dynamics for Monotone Inclusions , 2020, Set-Valued and Variational Analysis.

[28]  Michael I. Jordan,et al.  On dissipative symplectic integration with applications to gradient-based optimization , 2020, Journal of Statistical Mechanics: Theory and Experiment.

[29]  Nesterov Yurii,et al.  Inexact accelerated high-order proximal-point methods , 2020, Mathematical Programming.

[30]  Hedy Attouch,et al.  Newton-like Inertial Dynamics and Proximal Algorithms Governed by Maximally Monotone Operators , 2020, SIAM J. Optim..

[31]  Othmane Sebbouh,et al.  Convergence Rates of Damped Inertial Dynamics under Geometric Conditions and Perturbations , 2020, SIAM J. Optim..

[32]  Hedy Attouch,et al.  Convergence of a relaxed inertial proximal algorithm for maximally monotone operators , 2019, Mathematical Programming.

[33]  Daniel P. Robinson,et al.  Conformal symplectic and relativistic optimization , 2019, NeurIPS.

[34]  Aryan Mokhtari,et al.  A Unified Analysis of Extra-gradient and Optimistic Gradient Methods for Saddle Point Problems: Proximal Point Approach , 2019, AISTATS.

[35]  Bo Jiang,et al.  A Unified Adaptive Tensor Approximation Scheme to Accelerate Composite Convex Optimization , 2020, SIAM J. Optim..

[36]  H. Attouch,et al.  Fast convex optimization via time scaling of damped inertial gradient dynamics , 2020 .

[37]  Hedy Attouch,et al.  Fast Proximal Methods via Time Scaling of Damped Inertial Dynamics , 2019, SIAM J. Optim..

[38]  Yin Tat Lee,et al.  Near Optimal Methods for Minimizing Convex Functions with Lipschitz $p$-th Derivatives , 2019, Annual Conference Computational Learning Theory.

[39]  Brendan O'Donoghue,et al.  Hamiltonian descent for composite objectives , 2019, NeurIPS.

[40]  Michael I. Jordan,et al.  A Dynamical Systems Perspective on Nesterov Acceleration , 2019, ICML.

[41]  Andre Wibisono,et al.  Accelerating Rescaled Gradient Descent: Fast Optimization of Smooth Functions , 2019, NeurIPS.

[42]  S. Bellavia,et al.  Adaptive Regularization Algorithms with Inexact Evaluations for Nonconvex Optimization , 2018, SIAM J. Optim..

[43]  Jelena Diakonikolas,et al.  The Approximate Duality Gap Technique: A Unified Theory of First-Order Methods , 2017, SIAM J. Optim..

[44]  Zheng Qu,et al.  Adaptive restart of accelerated gradient methods under local quadratic growth condition , 2017, IMA Journal of Numerical Analysis.

[45]  H. Attouch,et al.  Rate of convergence of the Nesterov accelerated gradient method in the subcritical case α ≤ 3 , 2017, ESAIM: Control, Optimisation and Calculus of Variations.

[46]  Ohad Shamir,et al.  Oracle complexity of second-order methods for smooth convex optimization , 2017, Mathematical Programming.

[47]  Juan Peypouquet,et al.  Convergence of inertial dynamics and proximal algorithms governed by maximally monotone operators , 2017, Mathematical Programming.

[48]  John C. Duchi,et al.  Gradient Descent Finds the Cubic-Regularized Nonconvex Newton Step , 2016, SIAM J. Optim..

[49]  Yurii Nesterov,et al.  Lectures on Convex Optimization , 2018 .

[50]  H. Attouch,et al.  Convergence of damped inertial dynamics governed by regularized maximally monotone operators , 2018, Journal of Differential Equations.

[51]  Aryan Mokhtari,et al.  Direct Runge-Kutta Discretization Achieves Acceleration , 2018, NeurIPS.

[52]  Juan Peypouquet,et al.  Fast convergence of inertial dynamics and algorithms with asymptotic vanishing viscosity , 2018, Math. Program..

[53]  Jean-François Aujol,et al.  The Differential Inclusion Modeling FISTA Algorithm and Optimality of Convergence Rate in the Case b $\leq3$ , 2018, SIAM J. Optim..

[54]  Constantinos Daskalakis,et al.  Training GANs with Optimism , 2017, ICLR.

[55]  Yurii Nesterov,et al.  Relatively Smooth Convex Optimization by First-Order Methods, and Applications , 2016, SIAM J. Optim..

[56]  Daniel Kuhn,et al.  Data-driven distributionally robust optimization using the Wasserstein metric: performance guarantees and tractable reformulations , 2015, Mathematical Programming.

[57]  Florian A. Potra,et al.  A Superquadratic Variant of Newton's Method , 2017, SIAM J. Numer. Anal..

[58]  Hedy Attouch,et al.  Asymptotic stabilization of inertial gradient dynamics with time-dependent viscosity , 2017 .

[59]  José Mario Martínez,et al.  Worst-case evaluation complexity for unconstrained nonlinear optimization using high-order regularized models , 2017, Math. Program..

[60]  Alexandre d'Aspremont,et al.  Sharpness, Restart and Acceleration , 2017 .

[61]  Andre Wibisono,et al.  A variational perspective on accelerated methods in optimization , 2016, Proceedings of the National Academy of Sciences.

[62]  H. Attouch,et al.  Fast convex optimization via inertial dynamics with Hessian driven damping , 2016, Journal of Differential Equations.

[63]  Hedy Attouch,et al.  The Rate of Convergence of Nesterov's Accelerated Forward-Backward Method is Actually Faster Than 1/k2 , 2015, SIAM J. Optim..

[64]  Radu Ioan Bot,et al.  Second Order Forward-Backward Dynamical Systems For Monotone Inclusion Problems , 2015, SIAM J. Control. Optim..

[65]  Stephen P. Boyd,et al.  A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights , 2014, J. Mach. Learn. Res..

[66]  B. Svaiter,et al.  A dynamic approach to a proximal-Newton method for monotone inclusions in Hilbert spaces, with complexity O(1/n^2) , 2015, 1502.04286.

[67]  Emmanuel J. Candès,et al.  Adaptive Restart for Accelerated Gradient Schemes , 2012, Foundations of Computational Mathematics.

[68]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[69]  Benar Fux Svaiter,et al.  Newton-Like Dynamics and Forward-Backward Methods for Structured Monotone Inclusions in Hilbert Spaces , 2014, J. Optim. Theory Appl..

[70]  Simon Wotherspoon,et al.  A simple modification of Newton's method to achieve convergence of order 1 + √2 , 2014, Appl. Math. Lett..

[71]  Bruce Bueno de Mesquita,et al.  An Introduction to Game Theory , 2014 .

[72]  Renato D. C. Monteiro,et al.  An Accelerated Hybrid Proximal Extragradient Method for Convex Optimization and Its Implications to Second-Order Methods , 2013, SIAM J. Optim..

[73]  Benar Fux Svaiter,et al.  Global Convergence of a Closed-Loop Regularized Newton Method for Solving Monotone Inclusions in Hilbert Spaces , 2013, J. Optim. Theory Appl..

[74]  Paul-Emile Maingé First-Order Continuous Newton-like Systems for Monotone Inclusions , 2013, SIAM J. Control. Optim..

[75]  Yurii Nesterov,et al.  Gradient methods for minimizing composite functions , 2012, Mathematical Programming.

[76]  Renato D. C. Monteiro,et al.  Iteration-Complexity of a Newton Proximal Extragradient Method for Monotone Variational Inequalities and Inclusion Problems , 2012, SIAM J. Optim..

[77]  H. Attouch,et al.  A second-order differential system with hessian-driven damping; application to non-elastic shock laws , 2012 .

[78]  Benar Fux Svaiter,et al.  A Continuous Dynamical Newton-Like Approach to Solving Monotone Inclusions , 2011, SIAM J. Control. Optim..

[79]  Gonzalo Mateos,et al.  Distributed Sparse Linear Regression , 2010, IEEE Transactions on Signal Processing.

[80]  Renato D. C. Monteiro,et al.  On the Complexity of the Hybrid Proximal Extragradient Method for the Iterates and the Ergodic Mean , 2010, SIAM J. Optim..

[81]  Nicholas I. M. Gould,et al.  On solving trust-region and other regularised subproblems in optimization , 2010, Math. Program. Comput..

[82]  H. Attouch,et al.  ASYMPTOTIC BEHAVIOR OF SECOND-ORDER DISSIPATIVE EVOLUTION EQUATIONS COMBINING POTENTIAL WITH NON-POTENTIAL EFFECTS ∗ , 2009, 0905.0092.

[83]  Shie Mannor,et al.  Robustness and Regularization of Support Vector Machines , 2008, J. Mach. Learn. Res..

[84]  Marc Teboulle,et al.  A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , 2009, SIAM J. Imaging Sci..

[85]  Yurii Nesterov,et al.  Accelerating the cubic regularization of Newton’s method on convex problems , 2005, Math. Program..

[86]  Alicia Cordero,et al.  Variants of Newton's Method using fifth-order quadrature formulas , 2007, Appl. Math. Comput..

[87]  Ali Barati,et al.  A third-order Newton-type method to solve systems of nonlinear equations , 2007, Appl. Math. Comput..

[88]  Yurii Nesterov,et al.  Dual extrapolation and its applications to solving variational inequalities and related problems , 2003, Math. Program..

[89]  Jorge Cortés,et al.  Finite-time convergent gradient flows with applications to network consensus , 2006, Autom..

[90]  Yurii Nesterov,et al.  Cubic regularization of Newton method and its global performance , 2006, Math. Program..

[91]  Gábor Lugosi,et al.  Prediction, learning, and games , 2006 .

[92]  V. Stahl,et al.  Safe starting regions by fixed points and tightening , 1994, Computing.

[93]  L. Ambrosio,et al.  Gradient Flows: In Metric Spaces and in the Space of Probability Measures , 2005 .

[94]  Herbert H. H. Homeier A modified Newton method with cubic convergence: the multivariate case , 2004 .

[95]  M. Frontini,et al.  Third-order methods from quadrature formulae for solving systems of nonlinear equations , 2004, Appl. Math. Comput..

[96]  A DYNAMICAL SYSTEM ASSOCIATED WITH NEWTON ’ S METHOD FOR PARAMETRIC APPROXIMATIONS OF CONVEX MINIMIZATION PROBLEMS , 2004 .

[97]  Arkadi Nemirovski,et al.  Prox-Method with Rate of Convergence O(1/t) for Variational Inequalities with Lipschitz Continuous Monotone Operators and Smooth Convex-Concave Saddle Point Problems , 2004, SIAM J. Optim..

[98]  F. Facchinei,et al.  Finite-Dimensional Variational Inequalities and Complementarity Problems , 2003 .

[99]  A. Antipin,et al.  MINIMIZATION OF CONVEX FUNCTIONS ON CONVEX SETS BY MEANS OF DIFFERENTIAL EQUATIONS , 2003 .

[100]  J. Bolte,et al.  A second-order gradient-like dissipative dynamical system with Hessian-driven damping.: Application to optimization and mechanics , 2002 .

[101]  Heinz H. Bauschke,et al.  A Weak-to-Strong Convergence Principle for Fejé-Monotone Methods in Hilbert Spaces , 2001, Math. Oper. Res..

[102]  H. Attouch,et al.  An Inertial Proximal Method for Maximal Monotone Operators via Discretization of a Nonlinear Oscillator with Damping , 2001 .

[103]  H. Attouch,et al.  The Second-order in Time Continuous Newton Method , 2001 .

[104]  Marios M. Polycarpou,et al.  Cooperative Control of Distributed Multi-Agent Systems , 2001 .

[105]  Felipe Alvarez,et al.  On the Minimizing Property of a Second Order Dissipative System in Hilbert Spaces , 2000, SIAM J. Control. Optim..

[106]  H. Attouch,et al.  THE HEAVY BALL WITH FRICTION METHOD, I. THE CONTINUOUS DYNAMICAL SYSTEM: GLOBAL EXPLORATION OF THE LOCAL MINIMA OF A REAL-VALUED FUNCTION BY ASYMPTOTIC ANALYSIS OF A DISSIPATIVE DYNAMICAL SYSTEM , 2000 .

[107]  J. Schropp,et al.  A dynamical systems approach to constrained minimization , 2000 .

[108]  Nicholas I. M. Gould,et al.  Solving the Trust-Region Subproblem using the Lanczos Method , 1999, SIAM J. Optim..

[109]  O. Nelles,et al.  An Introduction to Optimization , 1996, IEEE Antennas and Propagation Magazine.

[110]  Michael T. Heath,et al.  Scientific Computing: An Introductory Survey , 1996 .

[111]  H. Attouch,et al.  A Dynamical Approach to Convex Minimization Coupling Approximation with the Steepest Descent Method , 1996 .

[112]  B. Lemaire An asymptotical variational principle associated with the steepest descent method for a convex function. , 1996 .

[113]  Johannes Schropp,et al.  Using dynamical systems methods to solve minimization problems , 1995 .

[114]  J. Verschelde,et al.  Homotopies exploiting Newton polytopes for solving sparse polynomial systems , 1994 .

[115]  Osman Güer On the convergence of the proximal point algorithm for convex minimization , 1991 .

[116]  Alexander P. Morgan,et al.  Chemical equilibrium systems as numerical test problems , 1990, TOMS.

[117]  Aleksej F. Filippov,et al.  Differential Equations with Discontinuous Righthand Sides , 1988, Mathematics and Its Applications.

[118]  A. Morgan,et al.  Errata: Computing all solutions to polynomial systems using homotopy continuation , 1987 .

[119]  A. Morgan Solving Polynomial Systems Using Continuation for Engineering and Scientific Problems , 1987 .

[120]  C. Kelley Solving Nonlinear Equations with Newton's Method , 1987 .

[121]  C. Kelley Iterative Methods for Linear and Nonlinear Equations , 1987 .

[122]  Shankar Sastry,et al.  A calculus for computing Filippov's differential inclusion with application to the variable structure control of robot manipulators , 1986, 1986 25th IEEE Conference on Decision and Control.

[123]  A. Nemirovskii,et al.  Optimal methods of smooth convex minimization , 1986 .

[124]  John E. Dennis,et al.  Numerical methods for unconstrained optimization and nonlinear equations , 1983, Prentice Hall series in computational mathematics.

[125]  Y. Nesterov A method for solving the convex programming problem with convergence rate O(1/k^2) , 1983 .

[126]  J. Traub Iterative Methods for the Solution of Equations , 1982 .

[127]  L. Popov A modification of the Arrow-Hurwicz method for search of saddle points , 1980 .

[128]  J. Baillon,et al.  Un exemple concernant le comportement asymptotique de la solution du problème dudt + ∂ϑ(μ) ∋ 0 , 1978 .

[129]  P. Lions,et al.  Une methode iterative de resolution d’une inequation variationnelle , 1978 .

[130]  H. Brezis Asymptotic Behavior of Some Evolution Systems , 1978 .

[131]  Ronald E. Bruck On the weak convergence of an ergodic iteration for the solution of variational inequalities for monotone operators in Hilbert space , 1977 .

[132]  R. Rockafellar Monotone Operators and the Proximal Point Algorithm , 1976 .

[133]  G. M. Korpelevich The extragradient method for finding saddle points and other problems , 1976 .

[134]  H. Brezis Opérateurs maximaux monotones et semi-groupes de contractions dans les espaces de Hilbert , 1973 .

[135]  James M. Ortega,et al.  Iterative solution of nonlinear equations in several variables , 2014, Computer science and applied mathematics.

[136]  J. Moreau Proximité et dualité dans un espace hilbertien , 1965 .

[137]  Boris Polyak Some methods of speeding up the convergence of iteration methods , 1964 .

[138]  D. E. Muller A method for solving algebraic equations using an automatic computer , 1956 .

[139]  E. Rowland Theory of Games and Economic Behavior , 1946, Nature.

[140]  R. Courant Variational methods for the solution of problems of equilibrium and vibrations , 1943 .