On the worst case performance of the steepest descent algorithm for quadratic functions

The existing choices for the step lengths used by the classical steepest descent algorithm for minimizing a convex quadratic function require in the worst case $$ \mathcal{{O}}(C\log (1/\varepsilon )) $$O(Clog(1/ε)) iterations to achieve a precision $$ \varepsilon $$ε, where C is the Hessian condition number. We show how to construct a sequence of step lengths with which the algorithm stops in $$ \mathcal{{O}}(\sqrt{C}\log (1/\varepsilon )) $$O(Clog(1/ε)) iterations, with a bound almost exactly equal to that of the Conjugate Gradient method.

[1]  George E. Forsythe.  n XS 61 fH CO o CD ON THE ASYMPTOTIC DIRECTIONS OF THE S-DIMENSIONAL OPTIMUM GRADIENT METHOD , 2015 .

[2]  John Darzentas,et al.  Problem Complexity and Method Efficiency in Optimization , 1983 .

[3]  Clóvis C. Gonzaga,et al.  On the steepest descent algorithm for quadratic functions , 2016, Comput. Optim. Appl..

[4]  J. Shewchuk.  An Introduction to the Conjugate Gradient Method Without the Agonizing Pain , 1994 .

[5]  William W. Hager,et al.  An efficient gradient method using the Yuan steplength , 2014, Comput. Optim. Appl..

[6]  G. Toraldo,et al.  On spectral properties of steepest descent methods , 2013 .

[7]  Marcos Raydan,et al.  Relaxed Steepest Descent and Cauchy-Barzilai-Borwein Method , 2002, Comput. Optim. Appl..

[8]  Werner L. Frank.  Solution of Linear Systems by Richardson's Method , 1960, JACM.

[9]  O. Nelles,et al.  An Introduction to Optimization , 1996, IEEE Antennas and Propagation Magazine.

[10]  T. J. Rivlin.  The Chebyshev polynomials , 1974 .

[11]  Yuhong Dai.  Alternate step gradient method , 2003 .

[12]  Marcos Raydan,et al.  The Barzilai and Borwein Gradient Method for the Large Scale Unconstrained Minimization Problem , 1997, SIAM J. Optim..

[13]  H. Akaike.  On a successive transformation of probability distribution and its application to the analysis of the optimum gradient method , 1959 .

[14]  Jorge Nocedal,et al.  On the Behavior of the Gradient Norm in the Steepest Descent Method , 2002, Comput. Optim. Appl..

[15]  Augustin-Louis Cauchy,et al.  ANALYSE MATHÉMATIQUE. – Méthodc générale pour la résolution des systèmes d'équations simultanées , 2009 .

[16]  J. Borwein,et al.  Two-Point Step Size Gradient Methods , 1988 .

[17]  J. Dicapua.  Chebyshev Polynomials , 2019, Fibonacci and Lucas Numbers With Applications.

[18]  Elizabeth W. Karas,et al.  COMPLEXITY OF FIRST-ORDER METHODS FOR DIFFERENTIABLE CONVEX OPTIMIZATION , 2014 .