A Nonlinear Conjugate Gradient Algorithm with an Optimal Property and an Improved Wolfe Line Search

In this paper, we seek the conjugate gradient direction closest to the direction of the scaled memoryless BFGS method and propose a family of conjugate gradient methods for unconstrained optimization. An improved Wolfe line search is also proposed, which can avoid a numerical drawback of the original Wolfe line search and guarantee the global convergence of the conjugate gradient method under mild conditions. To accelerate the algorithm, we introduce adaptive restarts along negative gradients based on the extent to which the function approximates some quadratic function during previous iterations. Numerical experiments with the CUTEr collection show that the proposed algorithm is promising.

[1]  M. Hestenes,et al.  Methods of conjugate gradients for solving linear systems , 1952 .

[2]  C. M. Reeves,et al.  Function minimization by conjugate gradients , 1964, Comput. J..

[3]  P. Wolfe.  Convergence Conditions for Ascent Methods. II , 1969 .

[4]  Boris Polyak.  The conjugate gradient method in extremal problems , 1969 .

[5]  E. Polak,et al.  Note sur la convergence de méthodes de directions conjuguées , 1969 .

[6]  P. Wolfe.  Convergence Conditions for Ascent Methods. II: Some Corrections , 1971 .

[7]  Shmuel S. Oren,et al.  Self-scaling variable metric algorithms for unconstrained minimization , 1972 .

[8]  D. Luenberger,et al.  Self-Scaling Variable Metric (SSVM) Algorithms , 1974 .

[9]  S. Oren.  SELF-SCALING VARIABLE METRIC (SSVM) ALGORITHMS Part II: Implementation and Experiments*t , 1974 .

[10]  Shmuel S. Oren,et al.  Optimal conditioning of self-scaling variable Metric algorithms , 1976, Math. Program..

[11]  M. J. D. Powell,et al.  Restart procedures for the conjugate gradient method , 1977, Math. Program..

[12]  A. Perry.  A Class of Conjugate Gradient Algorithms with a Two-Step Variable Metric Memory , 1977 .

[13]  D. Shanno.  On the Convergence of a New Conjugate Gradient Algorithm , 1978 .

[14]  D. F. Shanno,et al.  Matrix conditioning and nonlinear optimization , 1978, Math. Program..

[15]  T. M. Williams,et al.  Practical Methods of Optimization. Vol. 1: Unconstrained Optimization , 1980 .

[16]  David F. Shanno,et al.  Remark on “Algorithm 500: Minimization of Unconstrained Multivariate Functions [E4]” , 1980, TOMS.

[17]  Jorge J. Moré,et al.  Testing Unconstrained Optimization Software , 1981, TOMS.

[18]  M. Powell.  Nonconvex minimization calculations and the conjugate gradient method , 1984 .

[19]  M. Powell.  Convergence properties of algorithms for nonlinear optimization , 1986 .

[20]  R. Fletcher.  Practical Methods of Optimization , 1988 .

[21]  Jorge Nocedal,et al.  Global Convergence Properties of Conjugate Gradient Methods for Optimization , 1992, SIAM J. Optim..

[22]  Dimitri P. Bertsekas,et al.  Nonlinear Programming , 1997 .

[23]  M. Al-Baali.  Numerical Experience with a Class of Self-Scaling Quasi-Newton Algorithms , 1998 .

[24]  Stephen J. Wright,et al.  Numerical Optimization , 2018, Fundamental Statistical Inference.

[25]  Ya-Xiang Yuan,et al.  A Nonlinear Conjugate Gradient Method with a Strong Global Convergence Property , 1999, SIAM J. Optim..

[26]  Ya-Xiang Yuan,et al.  Convergence Properties of Nonlinear Conjugate Gradient Methods , 1999, SIAM J. Optim..

[27]  Y. -H. Dai,et al.  New Conjugacy Conditions and Related Nonlinear Conjugate Gradient Methods , 2001 .

[28]  Ya-Xiang Yuan,et al.  An Efficient Hybrid Conjugate Gradient Method for Unconstrained Optimization , 2001, Ann. Oper. Res..

[29]  Jorge J. Moré,et al.  Benchmarking optimization software with performance profiles , 2001, Math. Program..

[30]  Yu-Hong Dai.  A family of hybrid conjugate gradient methods for unconstrained optimization , 2003, Math. Comput..

[31]  Nicholas I. M. Gould,et al.  CUTEr and SifDec: A constrained and unconstrained testing environment, revisited , 2003, TOMS.

[32]  Hongchao Zhang,et al.  Adaptive Two-Point Stepsize Gradient Algorithm , 2001, Numerical Algorithms.

[33]  Hiroshi Yabe,et al.  Global Convergence Properties of Nonlinear Conjugate Gradient Methods with Modified Secant Condition , 2004, Comput. Optim. Appl..

[34]  William W. Hager,et al.  A New Conjugate Gradient Method with Guaranteed Descent and an Efficient Line Search , 2005, SIAM J. Optim..

[35]  W. Hager,et al.  A SURVEY OF NONLINEAR CONJUGATE GRADIENT METHODS , 2005 .

[36]  William W. Hager,et al.  Algorithm 851: CG_DESCENT, a conjugate gradient method with guaranteed descent , 2006, TOMS.

[37]  Li Zhang,et al.  Global convergence of a modified Fletcher–Reeves conjugate gradient method with Armijo-type line search , 2006, Numerische Mathematik.

[38]  Guoyin Li,et al.  New conjugacy condition and related new conjugate gradient methods for unconstrained optimization , 2007 .

[39]  D. Luenberger,et al.  SELF-SCALING VARIABLE METRIC ( SSVM ) ALGORITHMS Part I : Criteria and Sufficient Conditions for Scaling a Class of Algorithms * t , 2007 .

[40]  Wufan Chen,et al.  Spectral conjugate gradient methods with sufficient descent property for large-scale unconstrained optimization , 2008, Optim. Methods Softw..

[41]  Qunfeng Liu,et al.  Sufficient descent nonlinear conjugate gradient methods with conjugacy condition , 2009, Numerical Algorithms.

[42]  Yuhong Dai.  Nonlinear Conjugate Gradient Methods , 2011 .

[43]  Boris Polyak.  The conjugate gradient method in extreme problems , 2015 .

[44]  K. Schittkowski,et al.  NONLINEAR PROGRAMMING , 2022 .