A family of spectral gradient methods for optimization

We propose a family of spectral gradient methods, whose stepsize is determined by a convex combination of the long Barzilai–Borwein (BB) stepsize and the short BB stepsize. Each member of the family is shown to share certain quasi-Newton property in the sense of least squares. The family also includes some other gradient methods as its special cases. We prove that the family of methods is R-superlinearly convergent for two-dimensional strictly convex quadratics. Moreover, the family is R-linearly convergent in the any-dimensional case. Numerical results of the family with different settings are presented, which demonstrate that the proposed family is promising.

[1]  H. Akaike.  On a successive transformation of probability distribution and its application to the analysis of the optimum gradient method , 1959 .

[2]  C. G. Broyden.  A Class of Methods for Solving Nonlinear Simultaneous Equations , 1965 .

[3]  J. J. Moré,et al.  Quasi-Newton Methods, Motivation and Theory , 1974 .

[4]  L. Grippo,et al.  A nonmonotone line search technique for Newton's method , 1986 .

[5]  J. Borwein,et al.  Two-Point Step Size Gradient Methods , 1988 .

[6]  M. Raydan.  On the Barzilai and Borwein choice of steplength for the gradient method , 1993 .

[7]  Marcos Raydan,et al.  The Barzilai and Borwein Gradient Method for the Large Scale Unconstrained Minimization Problem , 1997, SIAM J. Optim..

[8]  J. M. Martínez,et al.  Gradient Method with Retards and Generalizations , 1998 .

[9]  José Mario Martínez,et al.  Nonmonotone Spectral Projected Gradient Methods on Convex Sets , 1999, SIAM J. Optim..

[10]  Jorge J. Moré,et al.  Benchmarking optimization software with performance profiles , 2001, Math. Program..

[11]  Jorge Nocedal,et al.  On the Behavior of the Gradient Norm in the Steepest Descent Method , 2002, Comput. Optim. Appl..

[12]  Marcos Raydan,et al.  Relaxed Steepest Descent and Cauchy-Barzilai-Borwein Method , 2002, Comput. Optim. Appl..

[13]  L. Liao,et al.  R-linear convergence of the Barzilai and Borwein gradient method , 2002 .

[14]  Yuhong Dai.  Alternate step gradient method , 2003 .

[15]  Ya-Xiang Yuan,et al.  Alternate minimization gradient method , 2003 .

[16]  Roger Fletcher,et al.  Projected Barzilai-Borwein methods for large-scale box-constrained quadratic programming , 2005, Numerische Mathematik.

[17]  Roger Fletcher,et al.  On the Barzilai-Borwein Method , 2005 .

[18]  Ya-Xiang Yuan,et al.  Analysis of monotone gradient methods , 2005 .

[19]  Roger Fletcher,et al.  On the asymptotic behaviour of some new gradient methods , 2005, Math. Program..

[20]  W. Hager,et al.  The cyclic Barzilai-–Borwein method for unconstrained optimization , 2006 .

[21]  Bin Zhou,et al.  Gradient Methods with Adaptive Step-Sizes , 2006, Comput. Optim. Appl..

[22]  Shiqian Ma,et al.  Projected Barzilai–Borwein method for large-scale nonnegative image restoration , 2007 .

[23]  L. Zanni,et al.  New adaptive stepsize selections in gradient methods , 2008 .

[24]  Zhouping Xin,et al.  Step-sizes for the gradient method , 2008 .

[25]  Stephen J. Wright,et al.  Sparse reconstruction by separable approximation , 2009, IEEE Trans. Signal Process..

[26]  Zhi-Quan Luo,et al.  Coordinated Beamforming for MISO Interference Channel: Complexity Analysis and Efficient Algorithms , 2011, IEEE Transactions on Signal Processing.

[27]  Yuhong Dai.  A New Analysis on the Barzilai-Borwein Gradient Method , 2013 .

[28]  Bo Jiang,et al.  Feasible Barzilai–Borwein-like methods for extreme symmetric eigenvalue problems , 2013, Optim. Methods Softw..

[29]  G. Toraldo,et al.  On spectral properties of steepest descent methods , 2013 .

[30]  William W. Hager,et al.  An efficient gradient method using the Yuan steplength , 2014, Comput. Optim. Appl..

[31]  Hongwei Liu,et al.  Quadratic regularization projected Barzilai–Borwein method for nonnegative matrix factorization , 2014, Data Mining and Knowledge Discovery.

[32]  José Mario Martínez,et al.  Spectral Projected Gradient Methods: Review and Perspectives , 2014 .

[33]  M. Al-Baali,et al.  A Positive Barzilai–Borwein-Like Stepsize and an Extension for Symmetric Linear Systems , 2015 .

[34]  Hongwei Liu,et al.  Smoothing projected Barzilai–Borwein method for constrained non-Lipschitz optimization , 2016, Comput. Optim. Appl..

[35]  Shiqian Ma,et al.  Barzilai-Borwein Step Size for Stochastic Gradient Descent , 2016, NIPS.

[36]  Clóvis C. Gonzaga,et al.  On the steepest descent algorithm for quadratic functions , 2016, Comput. Optim. Appl..

[37]  Yuhong Dai,et al.  A Barzilai-Borwein conjugate gradient method , 2016 .

[38]  Zdeněk Kalousek,et al.  Steepest Descent Method with Random Step Lengths , 2017, Found. Comput. Math..

[39]  Valeria Ruggiero,et al.  On the steplength selection in gradient methods for unconstrained optimization , 2018, Appl. Math. Comput..