A hybrid conjugate gradient method based on the self-scaled memoryless BFGS update

In this work, we present a new conjugate gradient method adapting the approach of the hybridization of the conjugate gradient update parameters of DY and HS+ convexly, which is based on a quasi-Newton philosophy. The computation of the hybrization parameter is obtained by minimizing the distance between the hybrid conjugate gradient direction and the self-scaling memoryless BFGS direction. Our numerical experiments indicate that our proposed method is preferable and in general superior to classic conjugate gradient methods in terms of efficiency and robustness.

[1]  Jorge J. Moré,et al.  Benchmarking optimization software with performance profiles , 2001, Math. Program..

[2]  Emilio Spedicato,et al.  Broyden's quasi-Newton methods for a nonlinear system of equations and unconstrained optimization: a review and open problems , 2014, Optim. Methods Softw..

[3]  A. Perry.  A Class of Conjugate Gradient Algorithms with a Two-Step Variable Metric Memory , 1977 .

[4]  D K Smith,et al.  Numerical Optimization , 2001, J. Oper. Res. Soc..

[5]  Nicholas I. M. Gould,et al.  CUTE: constrained and unconstrained testing environment , 1995, TOMS.

[6]  C. Storey,et al.  Global convergence result for conjugate gradient methods , 1991 .

[7]  Reza Ghanbari,et al.  A class of adaptive Dai–Liao conjugate gradient methods based on the scaled memoryless BFGS update , 2017, 4OR.

[8]  Neculai Andrei,et al.  Another hybrid conjugate gradient algorithm for unconstrained optimization , 2008, Numerical Algorithms.

[9]  Jorge Nocedal,et al.  On the limited memory BFGS method for large scale optimization , 1989, Math. Program..

[10]  C. X. Kou,et al.  A Modified Self-Scaling Memoryless Broyden–Fletcher–Goldfarb–Shanno Method for Unconstrained Optimization , 2015, J. Optim. Theory Appl..

[11]  D. G. Sotiropoulos,et al.  A memoryless BFGS neural network training algorithm , 2009, 2009 7th IEEE International Conference on Industrial Informatics.

[12]  J. Nocedal.  Updating Quasi-Newton Matrices With Limited Storage , 1980 .

[13]  M. Hestenes,et al.  Methods of conjugate gradients for solving linear systems , 1952 .

[14]  Jorge Nocedal,et al.  Analysis of a self-scaling quasi-Newton method , 1993, Math. Program..

[15]  Duan Li,et al.  On Restart Procedures for the Conjugate Gradient Method , 2004, Numerical Algorithms.

[16]  D. Luenberger,et al.  Self-Scaling Variable Metric (SSVM) Algorithms , 1974 .

[17]  M. Powell.  Nonconvex minimization calculations and the conjugate gradient method , 1984 .

[18]  Ya-Xiang Yuan,et al.  A Nonlinear Conjugate Gradient Method with a Strong Global Convergence Property , 1999, SIAM J. Optim..

[19]  W. Hager,et al.  A SURVEY OF NONLINEAR CONJUGATE GRADIENT METHODS , 2005 .

[20]  M. Al-Baali.  Numerical Experience with a Class of Self-Scaling Quasi-Newton Algorithms , 1998 .

[21]  Mehiddin Al-Baali.  Global and Superlinear Convergence of a Restricted Class of Self-Scaling Methods with Inexact Line Searches, for Convex Functions , 1998, Comput. Optim. Appl..

[22]  Neculai Andrei,et al.  Scaled memoryless BFGS preconditioned conjugate gradient algorithm for unconstrained optimization , 2007, Optim. Methods Softw..

[23]  Jorge Nocedal,et al.  Global Convergence Properties of Conjugate Gradient Methods for Optimization , 1992, SIAM J. Optim..

[24]  E. Polak,et al.  Note sur la convergence de méthodes de directions conjuguées , 1969 .

[25]  N. Andrei.  Hybrid Conjugate Gradient Algorithm for Unconstrained Optimization , 2009 .

[26]  Yu-Hong Dai,et al.  A Nonlinear Conjugate Gradient Algorithm with an Optimal Property and an Improved Wolfe Line Search , 2013, SIAM J. Optim..

[27]  Reza Ghanbari,et al.  A hybridization of the Hestenes–Stiefel and Dai–Yuan conjugate gradient methods based on a least-squares approach , 2015, Optim. Methods Softw..

[28]  C. M. Reeves,et al.  Function minimization by conjugate gradients , 1964, Comput. J..

[29]  M. Al-Baali.  Descent Property and Global Convergence of the Fletcher—Reeves Method with Inexact Line Search , 1985 .

[30]  D. Shanno.  On the Convergence of a New Conjugate Gradient Algorithm , 1978 .

[31]  D. Touati-Ahmed,et al.  Efficient hybrid conjugate gradient techniques , 1990 .

[32]  Nezam Mahdavi-Amiri,et al.  Two effective hybrid conjugate gradient algorithms based on modified BFGS updates , 2011, Numerical Algorithms.

[33]  Nezam Mahdavi-Amiri,et al.  Two Modified Hybrid Conjugate Gradient Methods Based on a Hybrid Secant Equation , 2013 .

[34]  T. M. Williams,et al.  Practical Methods of Optimization. Vol. 1: Unconstrained Optimization , 1980 .

[35]  Neculai Andrei,et al.  Accelerated hybrid conjugate gradient algorithm with modified secant condition for unconstrained optimization , 2010, Numerical Algorithms.

[36]  William W. Hager,et al.  Algorithm 851: CG_DESCENT, a conjugate gradient method with guaranteed descent , 2006, TOMS.

[37]  William W. Hager,et al.  A New Conjugate Gradient Method with Guaranteed Descent and an Efficient Line Search , 2005, SIAM J. Optim..

[38]  Li Zhang,et al.  Two descent hybrid conjugate gradient methods for optimization , 2008 .

[39]  Shmuel S. Oren,et al.  Optimal conditioning of self-scaling variable Metric algorithms , 1976, Math. Program..

[40]  D. Luenberger,et al.  SELF-SCALING VARIABLE METRIC ( SSVM ) ALGORITHMS Part I : Criteria and Sufficient Conditions for Scaling a Class of Algorithms * t , 2007 .

[41]  Yuhong Dai.  Nonlinear Conjugate Gradient Methods , 2011 .

[42]  Reza Ghanbari,et al.  Two hybrid nonlinear conjugate gradient methods based on a modified secant equation , 2014 .

[43]  Shmuel S. Oren,et al.  Self-scaling variable metric algorithms for unconstrained minimization , 1972 .

[44]  C. Storey,et al.  Efficient generalized conjugate gradient algorithms, part 1: Theory , 1991 .

[45]  Ya-Xiang Yuan,et al.  An Efficient Hybrid Conjugate Gradient Method for Unconstrained Optimization , 2001, Ann. Oper. Res..