Another Conjugate Gradient Algorithm with Guaranteed Descent and Conjugacy Conditions for Large-scale Unconstrained Optimization

In this paper, we suggest another accelerated conjugate gradient algorithm for which both the descent and the conjugacy conditions are guaranteed. The search direction is selected as a linear combination of the gradient and the previous direction. The coefficients in this linear combination are selected in such a way that both the descent and the conjugacy condition are satisfied at every iteration. The algorithm introduces the modified Wolfe line search, in which the parameter in the second Wolfe condition is modified at every iteration. It is shown that both for uniformly convex functions and for general nonlinear functions, the algorithm with strong Wolfe line search generates directions bounded away from infinity. The algorithm uses an acceleration scheme modifying the step length in such a manner as to improve the reduction of the function values along the iterations. Numerical comparisons with some conjugate gradient algorithms using a set of 75 unconstrained optimization problems with different dimensions show that the computational scheme outperforms the known conjugate gradient algorithms like Hestenes and Stiefel; Polak, Ribière and Polyak; Dai and Yuan or the hybrid Dai and Yuan; CG_DESCENT with Wolfe line search, as well as the quasi-Newton L-BFGS.

[1]  Guoyin Li,et al.  Global convergence of the Polak-Ribière-Polyak conjugate gradient method with an Armijo-type inexact line search for nonconvex unconstrained optimization problems , 2008, Math. Comput..

[2]  Nicholas I. M. Gould,et al.  CUTE: constrained and unconstrained testing environment , 1995, TOMS.

[3]  J. M. Martínez,et al.  A Spectral Conjugate Gradient Method for Unconstrained Optimization , 2001 .

[4]  Jorge Nocedal,et al.  On the limited memory BFGS method for large scale optimization , 1989, Math. Program..

[5]  Ya-Xiang Yuan,et al.  An Efficient Hybrid Conjugate Gradient Method for Unconstrained Optimization , 2001, Ann. Oper. Res..

[6]  Boris Polyak.  The conjugate gradient method in extremal problems , 1969 .

[7]  William W. Hager,et al.  Algorithm 851: CG_DESCENT, a conjugate gradient method with guaranteed descent , 2006, TOMS.

[8]  William W. Hager,et al.  A New Conjugate Gradient Method with Guaranteed Descent and an Efficient Line Search , 2005, SIAM J. Optim..

[9]  J. Craggs.  Applied Mathematical Sciences , 1973 .

[10]  R. Glowinski.  Lectures on Numerical Methods for Non-Linear Variational Problems , 1981 .

[11]  A. Perry.  A Modified Conjugate Gradient Algorithm for Unconstrained Nonlinear Optimization , 1975 .

[12]  M. Hestenes,et al.  Methods of conjugate gradients for solving linear systems , 1952 .

[13]  Jorge J. Moré,et al.  Digital Object Identifier (DOI) 10.1007/s101070100263 , 2001 .

[14]  Jerrold Bebernes,et al.  Mathematical Problems from Combustion Theory , 1989 .

[15]  M. J. D. Powell,et al.  Some convergence properties of the conjugate gradient method , 1976, Math. Program..

[16]  Guoyin Li,et al.  New conjugacy condition and related new conjugate gradient methods for unconstrained optimization , 2007 .

[17]  G. Cimatti.  On a problem of the theory of lubrication governed by a variational inequality , 1976 .

[18]  P. Wolfe.  Convergence Conditions for Ascent Methods. II , 1969 .

[19]  Neculai Andrei,et al.  An acceleration of gradient descent algorithm with backtracking for unconstrained optimization , 2006, Numerical Algorithms.

[20]  Dianne P. O'Leary,et al.  Conjugate Gradients and Related KMP Algorithms: The Beginnings , 1998 .

[21]  Guoliang Xue,et al.  The MINPACK-2 test problem collection , 1992 .

[22]  Gene H. Golub,et al.  Some History of the Conjugate Gradient and Lanczos Algorithms: 1948-1976 , 1989, SIAM Rev..

[23]  Neculai Andrei,et al.  Scaled memoryless BFGS preconditioned conjugate gradient algorithm for unconstrained optimization , 2007, Optim. Methods Softw..

[24]  Robert Osserman,et al.  Lectures on Minimal Surfaces. , 1991 .

[25]  Neculai Andrei.  A Numerical Study on Efficiency and Robustness of Some Conjugate Gradient Algorithms for Large-scale Unconstrained Optimization , 2013 .

[26]  David F. Shanno,et al.  Algorithm 500: Minimization of Unconstrained Multivariate Functions [E4] , 1976, TOMS.

[27]  Xiwen Lu,et al.  A conjugate gradient method with descent direction for unconstrained optimization , 2009, J. Comput. Appl. Math..

[28]  A. K. Kapila.  Review: Jerrold Bebernes and David Eberly, Mathematical problems from combustion theory , 1990 .

[29]  E. Polak,et al.  Note sur la convergence de méthodes de directions conjuguées , 1969 .

[30]  Ya-Xiang Yuan,et al.  Convergence Properties of Nonlinear Conjugate Gradient Methods , 1999, SIAM J. Optim..

[31]  P. Wolfe.  Convergence Conditions for Ascent Methods. II: Some Corrections , 1971 .

[32]  David F. Shanno,et al.  Conjugate Gradient Methods with Inexact Searches , 1978, Math. Oper. Res..

[33]  Duan Li,et al.  On Restart Procedures for the Conjugate Gradient Method , 2004, Numerical Algorithms.

[34]  Neculai Andrei,et al.  Acceleration of conjugate gradient algorithms for unconstrained optimization , 2009, Appl. Math. Comput..

[35]  E. Polak,et al.  Computational methods in optimization : a unified approach , 1972 .

[36]  David J. Thuente,et al.  Line search algorithms with guaranteed sufficient decrease , 1994, TOMS.

[37]  M. Powell.  Nonconvex minimization calculations and the conjugate gradient method , 1984 .

[38]  Jorge Nocedal.  Conjugate Gradient Methods and Nonlinear Optimization , 1996 .

[39]  Ya-Xiang Yuan,et al.  A Subspace Study on Conjugate Gradient Algorithms , 1995 .

[40]  Yu-Hong Dai.  New properties of a nonlinear conjugate gradient method , 2001, Numerische Mathematik.

[41]  Jorge Nocedal,et al.  Global Convergence Properties of Conjugate Gradient Methods for Optimization , 1992, SIAM J. Optim..

[42]  Avinoam Perry,et al.  Technical Note - A Modified Conjugate Gradient Algorithm , 1978, Oper. Res..

[43]  C. M. Reeves,et al.  Function minimization by conjugate gradients , 1964, Comput. J..

[44]  R. Aris.  The mathematical theory of diffusion and reaction in permeable catalysts. Volume II, Questions of uniqueness, stability, and transient behaviour , 1975 .

[45]  Neculai Andrei,et al.  A scaled BFGS preconditioned conjugate gradient algorithm for unconstrained optimization , 2007, Appl. Math. Lett..

[46]  Johannes C. C. Nitsche,et al.  Lectures on minimal surfaces: vol. 1 , 1989 .

[47]  W. Hager,et al.  A SURVEY OF NONLINEAR CONJUGATE GRADIENT METHODS , 2005 .

[48]  J. L. Nazareth,et al.  Linear and nonlinear conjugate gradient-related methods , 1996 .

[49]  Neculai Andrei,et al.  Scaled conjugate gradient algorithms for unconstrained optimization , 2007, Comput. Optim. Appl..

[50]  Neculai Andrei,et al.  An Unconstrained Optimization Test Functions Collection , 2008 .

[51]  Boris Polyak.  The conjugate gradient method in extreme problems , 2015 .

[52]  Gonglin Yuan,et al.  Modified nonlinear conjugate gradient methods with sufficient descent property for large-scale optimization problems , 2009, Optim. Lett..

[53]  R. Kohn,et al.  Numerical study of a relaxed variational problem from optimal design , 1986 .

[54]  Ya-Xiang Yuan.  Analysis on the conjugate gradient method , 1993 .

[55]  Y. -H. Dai,et al.  New Conjugacy Conditions and Related Nonlinear Conjugate Gradient Methods , 2001 .