Explicit Superlinear Convergence Rates of Broyden's Methods in Nonlinear Equations

In this paper, we study the explicit superlinear convergence rate of quasi-Newton methods. We particularly focus on the classical Broyden’s methods for solving nonlinear equations and establish their explicit (local) superlinear convergence rates when the initial point is close enough to a solution and the approximate Jacobian is close enough to the exact Jacobian related to the solution. Our results provide the explicit superlinear convergence rates of the Broyden’s “good” and “bad” methods for the first time. The explicit convergence rates provide some important insights on the performance difference between the “good” and “bad” methods. The theoretical findings in the convergence analysis of Broyden’s methods are also validated empirically in this paper.

[1]  H. Yabe,et al.  LOCAL AND SUPERLINEAR CONVERGENCE OF STRUCTURED QUASI-NEWTON METHODS FOR NONLINEAR OPTIMIZATION , 1996 .

[2]  Reinhold Schneider,et al.  An analysis for the DIIS acceleration method used in quantum chemistry calculations , 2011 .

[3]  C. G. Broyden.  The Convergence of a Class of Double-rank Minimization Algorithms 2. The New Algorithm , 1970 .

[4]  D. Shanno.  Conditioning of Quasi-Newton Methods for Function Minimization , 1970 .

[5]  John Greenstadt,et al.  On some classes of variationally derived Quasi-Newton methods for systems of nonlinear algebraic equations , 1978 .

[6]  A. Portela,et al.  ABS projection algorithms — Mathematical techniques for linear and nonlinear equations , 1992 .

[7]  A. Griewank.  The local convergence of Broyden-like methods on Lipschitzian problems in Hilbert spaces , 1987 .

[8]  C. T. Kelley,et al.  A New Proof of Superlinear Convergence for Broyden's Method in Hilbert Space , 1991, SIAM J. Optim..

[9]  P. Toint,et al.  Local convergence analysis for partitioned quasi-Newton updates , 1982 .

[10]  Y. Nesterov,et al.  Rates of superlinear convergence for classical quasi-Newton methods , 2020, Mathematical Programming.

[11]  Faster Explicit Superlinear Convergence for Greedy and Random Quasi-Newton Methods , 2021, 2104.08764.

[12]  C. G. Broyden.  Quasi-Newton methods and their application to function minimisation , 1967 .

[13]  D K Smith,et al.  Numerical Optimization , 2001, J. Oper. Res. Soc..

[14]  T. W. Mullikin.  Some probability distributions for neutron transport in a half-space , 1968 .

[15]  José Mario Martínez,et al.  Practical quasi-Newton methods for solving nonlinear systems , 2000 .

[16]  Lenhart K. Schubert.  Modification of a quasi-Newton method for nonlinear equations with a sparse Jacobian , 1970 .

[17]  Anton Rodomanov,et al.  Greedy Quasi-Newton Methods with Explicit Superlinear Convergence , 2020, SIAM J. Optim..

[18]  Stephen P. Boyd,et al.  Globally Convergent Type-I Anderson Acceleration for Nonsmooth Fixed-Point Iterations , 2018, SIAM J. Optim..

[19]  J. Sherman,et al.  Adjustment of an Inverse Matrix Corresponding to a Change in One Element of a Given Matrix , 1950 .

[20]  Aryan Mokhtari,et al.  Non-asymptotic superlinear convergence of standard quasi-Newton methods , 2020, Mathematical Programming.

[21]  John E. Dennis,et al.  On the Local and Superlinear Convergence of Quasi-Newton Methods , 1973 .

[22]  Shmuel S. Oren,et al.  Optimal conditioning of self-scaling variable Metric algorithms , 1976, Math. Program..

[23]  M. Powell.  On the Convergence of the Variable Metric Algorithm , 1971 .

[24]  Roger Fletcher,et al.  A Rapidly Convergent Descent Method for Minimization , 1963, Comput. J..

[25]  Emilio Spedicato,et al.  Broyden's quasi-Newton methods for a nonlinear system of equations and unconstrained optimization: a review and open problems , 2014, Optim. Methods Softw..

[26]  Ekkehard W. Sachs,et al.  Algorithmic methods in optimal control , 1981 .

[27]  Lucian-Liviu Albu,et al.  Non-Linear Models: Applications in Economics , 2006 .

[28]  Haishan Ye,et al.  Explicit Superlinear Convergence Rates of The SR1 Algorithm , 2021 .

[29]  Anton Rodomanov,et al.  New Results on Superlinear Convergence of Classical Quasi-Newton Methods , 2020, Journal of Optimization Theory and Applications.

[30]  Dmitry Kovalev,et al.  Fast Linear Convergence of Randomized BFGS , 2020, ArXiv.

[31]  C. G. Broyden.  A Class of Methods for Solving Nonlinear Simultaneous Equations , 1965 .

[32]  William C. Davidon,et al.  Variable Metric Method for Minimization , 1959, SIAM J. Optim..

[33]  C. G. Broyden.  The Convergence of a Class of Double-rank Minimization Algorithms 1. General Considerations , 1970 .

[34]  John E. Dennis,et al.  Numerical methods for unconstrained optimization and nonlinear equations , 1983, Prentice Hall series in computational mathematics.

[35]  C. Kelley.  Iterative Methods for Linear and Nonlinear Equations , 1987 .

[36]  L. Luksan,et al.  New quasi-Newton method for solving systems of nonlinear equations , 2017 .

[37]  R. Fletcher,et al.  A New Approach to Variable Metric Algorithms , 1970, Comput. J..

[38]  J. Nocedal,et al.  Global Convergence of a Class of Quasi-newton Methods on Convex Problems, Siam Some Global Convergence Properties of a Variable Metric Algorithm for Minimization without Exact Line Searches, Nonlinear Programming, Edited , 1996 .

[39]  D. M. Hwang,et al.  Convergence of Broyden's Method in Banach Spaces , 1991, SIAM J. Optim..

[40]  Defeng Sun,et al.  Newton and Quasi-Newton Methods for a Class of Nonsmooth Equations and Related Problems , 1997, SIAM J. Optim..

[41]  R. Schnabel,et al.  Solving Systems of Non-Linear Equations by Broyden&Apos;S Method with Projected Updates , 1977 .

[42]  D. Goldfarb.  A family of variable-metric methods derived by variational means , 1970 .

[43]  Andrzej STACHURSKI,et al.  Superlinear convergence of Broyden's boundedθ-class of methods , 1981, Math. Program..