Greedy Quasi-Newton Methods with Explicit Superlinear Convergence

In this paper, we study greedy variants of quasi-Newton methods. They are based on the updating formulas from a certain subclass of the Broyden family. In particular, this subclass includes the well-known DFP, BFGS and SR1 updates. However, in contrast to the classical quasi-Newton methods, which use the difference of successive iterates for updating the Hessian approximations, our methods apply basis vectors, greedily selected so as to maximize a certain measure of progress. For greedy quasi-Newton methods, we establish an explicit non-asymptotic bound on their rate of local superlinear convergence, which contains a contraction factor, depending on the square of the iteration counter. We also show that these methods produce Hessian approximations whose deviation from the exact Hessians linearly convergences to zero.

[1]  H. Yabe,et al.  LOCAL AND SUPERLINEAR CONVERGENCE OF STRUCTURED QUASI-NEWTON METHODS FOR NONLINEAR OPTIMIZATION , 1996 .

[2]  D. Shanno.  Conditioning of Quasi-Newton Methods for Function Minimization , 1970 .

[3]  Héctor J. Martínez,et al.  Local and Superlinear Convergence for Partially Known Quasi-Newton Methods , 1991, SIAM J. Optim..

[4]  Robert M. Gower,et al.  Randomized Quasi-Newton Updates Are Linearly Convergent Matrix Inversion Algorithms , 2016, SIAM J. Matrix Anal. Appl..

[5]  H. Yabe,et al.  Local and superlinear convergence of quasi-Newton methods based on modified secant conditions , 2007 .

[6]  Yurii Nesterov,et al.  Minimizing Uniformly Convex Functions by Cubic Regularization of Newton Method , 2019, Journal of Optimization Theory and Applications.

[7]  C. G. Broyden.  The Convergence of a Class of Double-rank Minimization Algorithms 2. The New Algorithm , 1970 .

[8]  Wenbo Gao,et al.  Quasi-Newton methods: superlinear convergence without line searches for self-concordant functions , 2016, Optim. Methods Softw..

[9]  J. J. Moré,et al.  Quasi-Newton Methods, Motivation and Theory , 1974 .

[10]  Aryan Mokhtari,et al.  IQN: An Incremental Quasi-Newton Method with Local Superlinear Convergence Rate , 2017, SIAM J. Optim..

[11]  C. G. Broyden.  The Convergence of a Class of Double-rank Minimization Algorithms 1. General Considerations , 1970 .

[12]  M. Powell.  On the Convergence of the Variable Metric Algorithm , 1971 .

[13]  Roger Fletcher,et al.  A Rapidly Convergent Descent Method for Minimization , 1963, Comput. J..

[14]  D K Smith,et al.  Numerical Optimization , 2001, J. Oper. Res. Soc..

[15]  R. Fletcher,et al.  A New Approach to Variable Metric Algorithms , 1970, Comput. J..

[16]  P. Toint,et al.  Local convergence analysis for partitioned quasi-Newton updates , 1982 .

[17]  C. G. Broyden.  Quasi-Newton methods and their application to function minimisation , 1967 .

[18]  Quoc Tran-Dinh,et al.  Generalized self-concordant functions: a recipe for Newton-type methods , 2017, Mathematical Programming.

[19]  Yurii Nesterov,et al.  Interior-point polynomial algorithms in convex programming , 1994, Siam studies in applied mathematics.

[20]  B. N. Pshenichnyi,et al.  Numerical Methods in Extremal Problems. , 1978 .

[21]  J. Nocedal,et al.  Global Convergence of a Class of Quasi-newton Methods on Convex Problems, Siam Some Global Convergence Properties of a Variable Metric Algorithm for Minimization without Exact Line Searches, Nonlinear Programming, Edited , 1996 .

[22]  Yurii Nesterov,et al.  Lectures on Convex Optimization , 2018 .

[23]  H. Walker.  Quasi-Newton Methods , 1978 .

[24]  J. J. Moré,et al.  A Characterization of Superlinear Convergence and its Application to Quasi-Newton Methods , 1973 .

[25]  D. Goldfarb.  A family of variable-metric methods derived by variational means , 1970 .

[26]  Andrzej STACHURSKI,et al.  Superlinear convergence of Broyden's boundedθ-class of methods , 1981, Math. Program..

[27]  J. Nocedal,et al.  A tool for the analysis of Quasi-Newton methods with application to unconstrained minimization , 1989 .

[28]  William C. Davidon,et al.  Variable Metric Method for Minimization , 1959, SIAM J. Optim..

[29]  Gonglin Yuan,et al.  The Superlinear Convergence of a Modified BFGS-Type Method for Unconstrained Optimization , 2004, Comput. Optim. Appl..

[30]  John E. Dennis,et al.  On the Local and Superlinear Convergence of Quasi-Newton Methods , 1973 .