A note on spectral properties of some gradient methods

Starting from the work by Barzilai and Borwein, gradient methods have gained a great amount of attention, and efficient low-cost schemes are available nowadays. The acceleration strategies used by these methods are based on the definition of effective steplength updating rules, which capture spectral properties of the Hessian of the objective function. The methods arising from this idea represent effective computational tools, extremely appealing for a variety of large-scale optimization problems arising in applications. In this work we discuss the spectral properties of some recently proposed gradient methods with the aim of providing insight into their computational effectiveness. Numerical experiments supporting and illustrating the theoretical analysis are provided.

[1]  Bin Zhou,et al.  Gradient Methods with Adaptive Step-Sizes , 2006, Comput. Optim. Appl..

[2]  L. Zanni,et al.  New adaptive stepsize selections in gradient methods , 2008 .

[3]  Luca Zanni,et al.  An Improved Gradient Projection-based Decomposition Technique for Support Vector Machines , 2006, Comput. Manag. Sci..

[4]  Roger Fletcher,et al.  On the Barzilai-Borwein Method , 2005 .

[5]  William W. Hager,et al.  An efficient gradient method using the Yuan steplength , 2014, Comput. Optim. Appl..

[6]  J. Borwein,et al.  Two-Point Step Size Gradient Methods , 1988 .

[7]  L. Zanni,et al.  Efficient deconvolution methods for astronomical imaging: algorithms and IDL-GPU codes , 2012, 1210.2258.

[8]  Daniela di Serafino,et al.  On the Application of the Spectral Projected Gradient Method in Image Segmentation , 2015, Journal of Mathematical Imaging and Vision.

[9]  Jorge Nocedal,et al.  On the Behavior of the Gradient Norm in the Steepest Descent Method , 2002, Comput. Optim. Appl..

[10]  Marcos Raydan,et al.  Relaxed Steepest Descent and Cauchy-Barzilai-Borwein Method , 2002, Comput. Optim. Appl..

[11]  Luca Zanni,et al.  Limited-memory scaled gradient projection methods for real-time image deconvolution in microscopy , 2014, Commun. Nonlinear Sci. Numer. Simul..

[12]  H. Akaike.  On a successive transformation of probability distribution and its application to the analysis of the optimum gradient method , 1959 .

[13]  W. Hager,et al.  The cyclic Barzilai-–Borwein method for unconstrained optimization , 2006 .

[14]  J. M. Martínez,et al.  Gradient Method with Retards and Generalizations , 1998 .

[15]  Yuhong Dai.  Alternate step gradient method , 2003 .

[16]  Clóvis C. Gonzaga,et al.  On the steepest descent algorithm for quadratic functions , 2016, Comput. Optim. Appl..

[17]  Daniela di Serafino,et al.  On the regularizing behavior of the SDA and SDC gradient methods in the solution of linear ill-posed problems , 2016, J. Comput. Appl. Math..

[18]  Roger Fletcher,et al.  A limited memory steepest descent method , 2011, Mathematical Programming.

[19]  G. Toraldo,et al.  On spectral properties of steepest descent methods , 2013 .