Greedy and Random Broyden's Methods with Explicit Superlinear Convergence Rates in Nonlinear Equations

In this paper, we propose the greedy and random Broyden’s method for solving nonlinear equations. Specifically, the greedy method greedily selects the direction to maximize a certain measure of progress for approximating the current Jacobian matrix, while the random method randomly chooses a direction. We establish explicit (local) superlinear convergence rates of both methods if the initial point and approximate Jacobian are close enough to a solution and corresponding Jacobian. Our two novel variants of Broyden’s method enjoy two important advantages that the approximate Jacobians of our algorithms will converge to the exact ones and the convergence rates of our algorithms are asymptotically faster than the original Broyden’s method. Our work is the first time to achieve such two advantages theoretically. Our experiments also empirically validate the advantages of our algorithms.

[1]  Vladlen Koltun,et al.  Multiscale Deep Equilibrium Models , 2020, NeurIPS.

[2]  C. G. Broyden.  The Convergence of a Class of Double-rank Minimization Algorithms 1. General Considerations , 1970 .

[3]  John E. Dennis,et al.  Numerical methods for unconstrained optimization and nonlinear equations , 1983, Prentice Hall series in computational mathematics.

[4]  C. G. Broyden.  A Class of Methods for Solving Nonlinear Simultaneous Equations , 1965 .

[5]  C. Kelley.  Iterative Methods for Linear and Nonlinear Equations , 1987 .

[6]  Aryan Mokhtari,et al.  Non-asymptotic superlinear convergence of standard quasi-Newton methods , 2020, Mathematical Programming.

[7]  R. Fletcher,et al.  A New Approach to Variable Metric Algorithms , 1970, Comput. J..

[8]  J. J. Moré,et al.  On the Global Convergence of Broyden''s Method , 1974 .

[9]  A. Griewank.  The local convergence of Broyden-like methods on Lipschitzian problems in Hilbert spaces , 1987 .

[10]  D K Smith,et al.  Numerical Optimization , 2001, J. Oper. Res. Soc..

[11]  Andreas Griewank,et al.  Broyden Updating, the Good and the Bad! , 2012 .

[12]  Anton Rodomanov,et al.  New Results on Superlinear Convergence of Classical Quasi-Newton Methods , 2020, Journal of Optimization Theory and Applications.

[13]  F. Soesianto,et al.  On the solution of highly structured nonlinear equations , 1992 .

[14]  Y. Nesterov,et al.  Rates of superlinear convergence for classical quasi-Newton methods , 2020, Mathematical Programming.

[15]  J. Sherman,et al.  Adjustment of an Inverse Matrix Corresponding to a Change in One Element of a Given Matrix , 1950 .

[16]  J. H. Avila,et al.  Update Methods for Highly Structured Systems of Nonlinear Equations , 1979 .

[17]  Anton Rodomanov,et al.  Greedy Quasi-Newton Methods with Explicit Superlinear Convergence , 2020, SIAM J. Optim..

[18]  R. Schnabel.  Quasi-Newton Methods Using Multiple Secant Equations. , 1983 .

[19]  D. M. Hwang,et al.  Convergence of Broyden's Method in Banach Spaces , 1991, SIAM J. Optim..

[20]  Haishan Ye,et al.  Explicit Superlinear Convergence of Broyden’s Method in Nonlinear Equations , 2021 .

[21]  William C. Davidon,et al.  Variable Metric Method for Minimization , 1959, SIAM J. Optim..

[22]  Oleg Burdakov,et al.  On Superlinear Convergence of Some Stable Variants of the Secant Method , 1986 .

[23]  Oleg Burdakov,et al.  Stable versions of the secants method for solving systems of equations , 1983 .

[24]  D. Goldfarb.  A family of variable-metric methods derived by variational means , 1970 .

[25]  Renpu Ge,et al.  The convergence of variable metric matrices in unconstrained optimization , 1983, Math. Program..

[26]  R. Schnabel.  Quasi-Newton Methods Using Multiple Secant Equations ; CU-CS-247-83 , 1983 .

[27]  José Mario Martínez,et al.  A Quasi-Newton method with modification of one column per iteration , 1984, Computing.

[28]  Benedetta Morini,et al.  Quasi-Newton methods for constrained nonlinear systems: complexity analysis and applications , 2018, Comput. Optim. Appl..

[29]  Vladlen Koltun,et al.  Deep Equilibrium Models , 2019, NeurIPS.

[30]  Roger Fletcher,et al.  A Rapidly Convergent Descent Method for Minimization , 1963, Comput. J..

[31]  Emilio Spedicato,et al.  Broyden's quasi-Newton methods for a nonlinear system of equations and unconstrained optimization: a review and open problems , 2014, Optim. Methods Softw..

[32]  Ekkehard W. Sachs,et al.  Algorithmic methods in optimal control , 1981 .

[33]  Lucian-Liviu Albu,et al.  Non-Linear Models: Applications in Economics , 2006 .

[34]  Florian Mannel,et al.  Convergence properties of the Broyden-like method for mixed linear–nonlinear systems of equations , 2021, Numerical Algorithms.

[35]  John E. Dennis,et al.  On the Local and Superlinear Convergence of Quasi-Newton Methods , 1973 .

[36]  José Mario Martínez,et al.  Practical quasi-Newton methods for solving nonlinear systems , 2000 .

[37]  John Greenstadt,et al.  On some classes of variationally derived Quasi-Newton methods for systems of nonlinear algebraic equations , 1978 .

[38]  Faster Explicit Superlinear Convergence for Greedy and Random Quasi-Newton Methods , 2021, 2104.08764.

[39]  C. G. Broyden.  Quasi-Newton methods and their application to function minimisation , 1967 .

[40]  D. Shanno.  Conditioning of Quasi-Newton Methods for Function Minimization , 1970 .

[41]  T. W. Mullikin.  Some probability distributions for neutron transport in a half-space , 1968 .

[42]  Marco A. López,et al.  Local convergence of quasi-Newton methods under metric regularity , 2014, Comput. Optim. Appl..

[43]  C. G. Broyden.  The Convergence of a Class of Double-rank Minimization Algorithms 2. The New Algorithm , 1970 .

[44]  J. J. Moré,et al.  Quasi-Newton Methods, Motivation and Theory , 1974 .

[45]  A. Portela,et al.  ABS projection algorithms — Mathematical techniques for linear and nonlinear equations , 1992 .

[46]  C. T. Kelley,et al.  A New Proof of Superlinear Convergence for Broyden's Method in Hilbert Space , 1991, SIAM J. Optim..