A Numerical Study on Efficiency and Robustness of Some Conjugate Gradient Algorithms for Large-scale Unconstrained Optimization

A numerical evaluation and comparisons using performance profiles of some representative conjugate gradient algorithms for solving a large variety of large-scale unconstrained optimization problems are carried on. In this intensive numerical study we selected eight known conjugate gradient algorithms: Hestenes and Stiefel (HS), Polak-RibièrePolyak (PRP), CONMIN, ASCALCG, CG-DESCENT, AHYBRIDM, THREECG and DESCON. These algorithms are different in many respects. However, they have a lot of concepts in common, which give the numerical comparisons sense and confident expectations. The initial search direction in all algorithms is the negative gradient computed in the initial point and the step length is computed by the Wolfe line search conditions. Excepting CONMIN and CG-DESCENT, all the algorithms from this numerical study implement an acceleration scheme which modifies the step length in a multiplicative manner to improve the reduction of the functions values along the iterations. The numerical study is based on a set of 800 artificially large-scale unconstrained optimization test functions of different complexity and with different structures of their Hessian matrix. A detailed numerical evaluation based on performance profiles is applied to the comparisons of these algorithms showing that all of them are able to solve difficult large-scale unconstrained optimization problems. However, comparisons using only artificially test problems are weak and dependent by arbitrary choices concerning the stopping criteria of the algorithms and on decision of whether an algorithm found a solution or not. To get definitive conclusions using this sort of comparisons based only on artificially test problems is an illusion. However, using some real unconstrained optimization applications we can get a more confident conclusion about the efficiency and robustness of optimization algorithms considered in this numerical study.

[1]  Duan Li,et al.  On Restart Procedures for the Conjugate Gradient Method , 2004, Numerical Algorithms.

[2]  Jerrold Bebernes,et al.  Mathematical Problems from Combustion Theory , 1989 .

[3]  Robert Osserman,et al.  Lectures on Minimal Surfaces. , 1991 .

[4]  Constanţa Zoie Rădulescu,et al.  A Decision Support Tool Based on a Portfolio Selection Model for Crop Planning under Risk , 2012 .

[5]  Neculai Andrei,et al.  Another Conjugate Gradient Algorithm with Guaranteed Descent and Conjugacy Conditions for Large-scale Unconstrained Optimization , 2013, J. Optim. Theory Appl..

[6]  Neculai Andrei,et al.  Acceleration of conjugate gradient algorithms for unconstrained optimization , 2009, Appl. Math. Comput..

[7]  M. Powell.  Nonconvex minimization calculations and the conjugate gradient method , 1984 .

[8]  T. M. Williams,et al.  Practical Methods of Optimization. Vol. 1: Unconstrained Optimization , 1980 .

[9]  Neculai Andrei,et al.  On three-term conjugate gradient algorithms for unconstrained optimization , 2013, Appl. Math. Comput..

[10]  Neculai Andrei,et al.  Accelerated hybrid conjugate gradient algorithm with modified secant condition for unconstrained optimization , 2010, Numerical Algorithms.

[11]  Jorge Nocedal,et al.  A Numerical Study of the Limited Memory BFGS Method and the Truncated-Newton Method for Large Scale Optimization , 1991, SIAM J. Optim..

[12]  Neculai Andrei,et al.  A Hybrid Conjugate Gradient Algorithm for Unconstrained Optimization as a Convex Combination of Hestenes-Stiefel and Dai-Yuan , 2008 .

[13]  R. Glowinski.  Lectures on Numerical Methods for Non-Linear Variational Problems , 1981 .

[14]  Guoliang Xue,et al.  The MINPACK-2 test problem collection , 1992 .

[15]  Neculai Andrei,et al.  Accelerated scaled memoryless BFGS preconditioned conjugate gradient algorithm for unconstrained optimization , 2010, Eur. J. Oper. Res..

[16]  Johannes C. C. Nitsche,et al.  Lectures on minimal surfaces: vol. 1 , 1989 .

[17]  Ya-Xiang Yuan,et al.  A Nonlinear Conjugate Gradient Method with a Strong Global Convergence Property , 1999, SIAM J. Optim..

[18]  Guoyin Li,et al.  New conjugacy condition and related new conjugate gradient methods for unconstrained optimization , 2007 .

[19]  D. Shanno.  On the Convergence of a New Conjugate Gradient Algorithm , 1978 .

[20]  D. Touati-Ahmed,et al.  Efficient hybrid conjugate gradient techniques , 1990 .

[21]  P. Toint,et al.  Partitioned variable metric updates for large structured optimization problems , 1982 .

[22]  C. Storey,et al.  Global convergence result for conjugate gradient methods , 1991 .

[23]  Neculai Andrei,et al.  An Unconstrained Optimization Test Functions Collection , 2008 .

[24]  Boris Polyak.  The conjugate gradient method in extreme problems , 2015 .

[25]  David F. Shanno,et al.  Algorithm 500: Minimization of Unconstrained Multivariate Functions [E4] , 1976, TOMS.

[26]  A. K. Kapila.  Review: Jerrold Bebernes and David Eberly, Mathematical problems from combustion theory , 1990 .

[27]  E. Polak,et al.  Note sur la convergence de méthodes de directions conjuguées , 1969 .

[28]  N. Andrei.  Hybrid Conjugate Gradient Algorithm for Unconstrained Optimization , 2009 .

[29]  J. M. Martínez,et al.  A Spectral Conjugate Gradient Method for Unconstrained Optimization , 2001 .

[30]  A. Perry.  A Class of Conjugate Gradient Algorithms with a Two-Step Variable Metric Memory , 1977 .

[31]  Neculai Andrei,et al.  Scaled conjugate gradient algorithms for unconstrained optimization , 2007, Comput. Optim. Appl..

[32]  Neculai Andrei,et al.  An acceleration of gradient descent algorithm with backtracking for unconstrained optimization , 2006, Numerical Algorithms.

[33]  David F. Shanno,et al.  Conjugate Gradient Methods with Inexact Searches , 1978, Math. Oper. Res..

[34]  J. Daniel.  The Conjugate Gradient Method for Linear and Nonlinear Operator Equations , 1967 .

[35]  R. Kohn,et al.  Numerical study of a relaxed variational problem from optimal design , 1986 .

[36]  Stefan M. Wild,et al.  Benchmarking Derivative-Free Optimization Algorithms , 2009, SIAM J. Optim..

[37]  G. Cimatti.  On a problem of the theory of lubrication governed by a variational inequality , 1976 .

[38]  P. Wolfe.  Convergence Conditions for Ascent Methods. II , 1969 .

[39]  C. M. Reeves,et al.  Function minimization by conjugate gradients , 1964, Comput. J..

[40]  R. Aris.  The mathematical theory of diffusion and reaction in permeable catalysts. Volume II, Questions of uniqueness, stability, and transient behaviour , 1975 .

[41]  Boris Polyak.  The conjugate gradient method in extremal problems , 1969 .

[42]  William W. Hager,et al.  A New Conjugate Gradient Method with Guaranteed Descent and an Efficient Line Search , 2005, SIAM J. Optim..

[43]  R. Fletcher.  Practical Methods of Optimization , 1988 .

[44]  M. Hestenes,et al.  Methods of conjugate gradients for solving linear systems , 1952 .

[45]  Jorge J. Moré,et al.  Digital Object Identifier (DOI) 10.1007/s101070100263 , 2001 .

[46]  Neculai Andrei,et al.  Another hybrid conjugate gradient algorithm for unconstrained optimization , 2008, Numerical Algorithms.

[47]  P. Wolfe.  Convergence Conditions for Ascent Methods. II: Some Corrections , 1971 .

[48]  Nicholas I. M. Gould,et al.  CUTE: constrained and unconstrained testing environment , 1995, TOMS.

[49]  Jorge Nocedal,et al.  Global Convergence Properties of Conjugate Gradient Methods for Optimization , 1992, SIAM J. Optim..

[50]  C. Storey,et al.  Efficient generalized conjugate gradient algorithms, part 1: Theory , 1991 .

[51]  M. J. D. Powell,et al.  Restart procedures for the conjugate gradient method , 1977, Math. Program..

[52]  Ya-Xiang Yuan,et al.  An Efficient Hybrid Conjugate Gradient Method for Unconstrained Optimization , 2001, Ann. Oper. Res..

[53]  Saman Babaie-Kafaki,et al.  A note on the global convergence theorem of the scaled conjugate gradient algorithms proposed by Andrei , 2012, Comput. Optim. Appl..

[54]  Neculai Andrei,et al.  A simple three-term conjugate gradient algorithm for unconstrained optimization , 2013, J. Comput. Appl. Math..