Nonlinear Conjugate Gradient Methods

Conjugate gradient methods are a class of important methods for solving linear equations and for solving nonlinear optimization. In this article, a review on conjugate gradient methods for unconstrained optimization is given. They are divided into early conjugate gradient methods, descent conjugate gradient methods, and sufficient descent conjugate gradient methods. Two general convergence theorems are provided for the conjugate gradient method assuming the descent property of each search direction. Some research issues on conjugate gradient methods are mentioned. Keywords: conjugate gradient method; line search; descent property; sufficient descent condition; global convergence

[1]  M. Hestenes,et al.  Methods of conjugate gradients for solving linear systems , 1952 .

[2]  C. M. Reeves,et al.  Function minimization by conjugate gradients , 1964, Comput. J..

[3]  L. Armijo.  Minimization of functions having Lipschitz continuous first partial derivatives. , 1966 .

[4]  P. Wolfe.  Convergence Conditions for Ascent Methods. II , 1969 .

[5]  Boris Polyak.  The conjugate gradient method in extremal problems , 1969 .

[6]  C. G. Broyden.  The Convergence of a Class of Double-rank Minimization Algorithms 1. General Considerations , 1970 .

[7]  P. Wolfe.  Convergence Conditions for Ascent Methods. II: Some Corrections , 1971 .

[8]  H. Crowder,et al.  Linear convergence of the conjugate gradient method , 1972 .

[9]  A. I. Cohen.  Rate of convergence of several conjugate gradient algorithms. , 1972 .

[10]  K. Ritter,et al.  Alternative proofs of the convergence properties of the conjugate-gradient method , 1974 .

[11]  M. J. D. Powell,et al.  Restart procedures for the conjugate gradient method , 1977, Math. Program..

[12]  A. Perry.  A Class of Conjugate Gradient Algorithms with a Two-Step Variable Metric Memory , 1977 .

[13]  L. Nazareth.  A conjugate direction algorithm without line searches , 1977 .

[14]  David F. Shanno,et al.  Conjugate Gradient Methods with Inexact Searches , 1978, Math. Oper. Res..

[15]  Albert G. Buckley,et al.  A combined conjugate-gradient quasi-Newton minimization algorithm , 1978, Math. Program..

[16]  T. M. Williams,et al.  Practical Methods of Optimization. Vol. 1: Unconstrained Optimization , 1980 .

[17]  Magnus R. Hestenes,et al.  Conjugate Direction Methods in Optimization , 1980 .

[18]  Claude Lemaréchal,et al.  A view of line-searches , 1981 .

[19]  Jorge J. Moré,et al.  Testing Unconstrained Optimization Software , 1981, TOMS.

[20]  Luigi Grippo,et al.  Stopping criteria for linesearch methods without derivatives , 1984, Math. Program..

[21]  M. Powell.  Nonconvex minimization calculations and the conjugate gradient method , 1984 .

[22]  M. Al-Baali.  Descent Property and Global Convergence of the Fletcher—Reeves Method with Inexact Line Search , 1985 .

[23]  L. Grippo,et al.  A nonmonotone line search technique for Newton's method , 1986 .

[24]  M. Powell.  Convergence properties of algorithms for nonlinear optimization , 1986 .

[25]  J. Nocedal,et al.  Global Convergence of a Class of Quasi-newton Methods on Convex Problems, Siam Some Global Convergence Properties of a Variable Metric Algorithm for Minimization without Exact Line Searches, Nonlinear Programming, Edited , 1996 .

[26]  R. Fletcher.  Practical Methods of Optimization , 1988 .

[27]  J. Borwein,et al.  Two-Point Step Size Gradient Methods , 1988 .

[28]  L. Grippo,et al.  Global convergence and stabilization of unconstrained minimization methods without derivatives , 1988 .

[29]  Jorge Nocedal,et al.  On the limited memory BFGS method for large scale optimization , 1989, Math. Program..

[30]  D. Touati-Ahmed,et al.  Efficient hybrid conjugate gradient techniques , 1990 .

[31]  D. Pu,et al.  On the convergence property of the DFP algorithm , 1990 .

[32]  C. Storey,et al.  Global convergence result for conjugate gradient methods , 1991 .

[33]  C. Storey,et al.  Efficient generalized conjugate gradient algorithms, part 1: Theory , 1991 .

[34]  Jorge Nocedal,et al.  Global Convergence Properties of Conjugate Gradient Methods for Optimization , 1992, SIAM J. Optim..

[35]  Jorge Nocedal,et al.  Theory of algorithms for unconstrained optimization , 1992, Acta Numerica.

[36]  David J. Thuente,et al.  Line search algorithms with guaranteed sufficient decrease , 1994, TOMS.

[37]  R. Pytlak.  On the convergence of conjugate gradient algorithms , 1994 .

[38]  Ya-Xiang Yuan,et al.  A Subspace Study on Conjugate Gradient Algorithms , 1995 .

[39]  Nicholas I. M. Gould,et al.  CUTE: constrained and unconstrained testing environment , 1995, TOMS.

[40]  Liu Guanghui,et al.  Global convergence of the fletcher-reeves algorithm with inexact linesearch , 1995 .

[41]  Ya-Xiang Yuan,et al.  Convergence properties of the Fletcher-Reeves method , 1996 .

[42]  Luigi Grippo,et al.  A globally convergent version of the Polak-Ribière conjugate gradient method , 1997, Math. Program..

[43]  Marcos Raydan,et al.  The Barzilai and Borwein Gradient Method for the Large Scale Unconstrained Minimization Problem , 1997, SIAM J. Optim..

[44]  Yuzhong Zhang,et al.  Global convergence property ofs-dependent GFR conjugate gradient method , 1998 .

[45]  Jingfeng Zhang,et al.  New Quasi-Newton Equation and Related Methods for Unconstrained Optimization , 1999 .

[46]  Stephen J. Wright,et al.  Conjugate Gradient Methods , 1999 .

[47]  Ya-Xiang Yuan,et al.  Global convergence of the method of shortest residuals , 1999, Numerische Mathematik.

[48]  Ya-Xiang Yuan,et al.  A Nonlinear Conjugate Gradient Method with a Strong Global Convergence Property , 1999, SIAM J. Optim..

[49]  Ya-Xiang Yuan,et al.  Convergence Properties of Nonlinear Conjugate Gradient Methods , 1999, SIAM J. Optim..

[50]  Jorge Nocedal,et al.  Automatic Preconditioning by Limited Memory Quasi-Newton Updating , 1999, SIAM J. Optim..

[51]  L. Liao,et al.  New Conjugacy Conditions and Related Nonlinear Conjugate Gradient Methods , 2001 .

[52]  Ya-Xiang Yuan,et al.  A three-parameter family of nonlinear conjugate gradient methods , 2001, Math. Comput..

[53]  J. M. Martínez,et al.  A Spectral Conjugate Gradient Method for Unconstrained Optimization , 2001 .

[54]  Yu-Hong Dai.  New properties of a nonlinear conjugate gradient method , 2001, Numerische Mathematik.

[55]  Jianzhon Zhang,et al.  Properties and numerical performance of quasi-Newton methods with modified quasi-Newton equations , 2001 .

[56]  Ya-Xiang Yuan,et al.  An Efficient Hybrid Conjugate Gradient Method for Unconstrained Optimization , 2001, Ann. Oper. Res..

[57]  Jie Sun,et al.  Global convergence of a two-parameter family of conjugate gradient methods without line search , 2002 .

[58]  DaiYuhong,et al.  A NONMONOTONE CONJUGATE GRADIENT ALGORITHM FOR UNCONSTRAINED OPTIMIZATION , 2002 .

[59]  L. Liao,et al.  R-linear convergence of the Barzilai and Borwein gradient method , 2002 .

[60]  Yu-Hong Dai.  A family of hybrid conjugate gradient methods for unconstrained optimization , 2003, Math. Comput..

[61]  Sun Linping.  Updating the Self-Scaling Symmetric Rank One Algorithm with Limited Memory for Large-Scale Unconstrained Optimization , 2004 .

[62]  Hiroshi Yabe,et al.  Global Convergence Properties of Nonlinear Conjugate Gradient Methods with Modified Secant Condition , 2004, Comput. Optim. Appl..

[63]  Gonglin Yuan,et al.  The Superlinear Convergence of a Modified BFGS-Type Method for Unconstrained Optimization , 2004, Comput. Optim. Appl..

[64]  William W. Hager,et al.  A New Conjugate Gradient Method with Guaranteed Descent and an Efficient Line Search , 2005, SIAM J. Optim..

[65]  W. Hager,et al.  A SURVEY OF NONLINEAR CONJUGATE GRADIENT METHODS , 2005 .

[66]  Weijun Zhou,et al.  A descent modified Polak–Ribière–Polyak conjugate gradient method and its global convergence , 2006 .

[67]  William W. Hager,et al.  Algorithm 851: CG_DESCENT, a conjugate gradient method with guaranteed descent , 2006, TOMS.

[68]  Li Zhang,et al.  Global convergence of a modified Fletcher–Reeves conjugate gradient method with Armijo-type line search , 2006, Numerische Mathematik.

[69]  Radosław Pytlak,et al.  On the Method of Shortest Residuals for Unconstrained Optimization , 2007 .

[70]  Guoyin Li,et al.  New conjugacy condition and related new conjugate gradient methods for unconstrained optimization , 2007 .

[71]  W. Cheng.  A Two-Term PRP-Based Descent Method , 2007 .

[72]  Wufan Chen,et al.  Spectral conjugate gradient methods with sufficient descent property for large-scale unconstrained optimization , 2008, Optim. Methods Softw..

[73]  Neculai Andrei,et al.  Accelerated hybrid conjugate gradient algorithm with modified secant condition for unconstrained optimization , 2010, Numerical Algorithms.

[74]  J. L. Nazareth,et al.  Conjugate gradient method , 2009 .

[75]  Qunfeng Liu,et al.  Sufficient descent nonlinear conjugate gradient methods with conjugacy condition , 2009, Numerical Algorithms.

[76]  Jinhua Guo,et al.  A new family of conjugate gradient methods , 2009 .

[77]  Neculai Andrei,et al.  Accelerated scaled memoryless BFGS preconditioned conjugate gradient algorithm for unconstrained optimization , 2010, Eur. J. Oper. Res..

[78]  Yuhong Dai.  Convergence Analysis of Nonlinear Conjugate Gradient Methods , 2010 .

[79]  Yu-Hong Dai.  Convergence of conjugate gradient methods with constant stepsizes , 2011, Optim. Methods Softw..