A double parameter scaled BFGS method for unconstrained optimization

Abstract A double parameter scaled BFGS method for unconstrained optimization is presented. In this method, the first two terms of the known BFGS update formula are scaled with a positive parameter while the third one is scaled with another positive parameter. These parameters are selected in such a way as to improve the eigenvalues structure of the BFGS update. The parameter scaling the first two terms of the BFGS update is determined by clustering the eigenvalues of the scaled BFGS matrix. On the other hand, the parameter scaling the third term is determined as a preconditioner to the Hessian of the minimizing function combined with the minimization of the conjugacy condition from conjugate gradient methods. Under the inexact Wolfe line search, the global convergence of the double parameter scaled BFGS method is proved in very general conditions without assuming the convexity of the minimizing function. Using 80 unconstrained optimization test functions with a medium number of variables, the preliminary numerical experiments show that this double parameter scaled BFGS method is more efficient than the standard BFGS update or than some other scaled BFGS methods.

[1]  M. J. D. Powell,et al.  Updating conjugate directions by the BFGS formula , 1987, Math. Program..

[2]  M. J. D. Powell,et al.  How bad are the BFGS and DFP methods when the objective function is quadratic? , 1986, Math. Program..

[3]  Andreas Griewank,et al.  The global convergence of partitioned BFGS on problems with convex decompositions and Lipschitzian gradients , 1991, Math. Program..

[4]  Ya-Xiang Yuan,et al.  A modified BFGS algorithm for unconstrained optimization , 1991 .

[5]  C. G. Broyden.  The Convergence of a Class of Double-rank Minimization Algorithms 2. The New Algorithm , 1970 .

[6]  M. Al-Baali.  Numerical Experience with a Class of Self-Scaling Quasi-Newton Algorithms , 1998 .

[7]  Aiping Liao,et al.  Modifying the BFGS method , 1997, Oper. Res. Lett..

[8]  M. C. Biggs.  Minimization Algorithms Making Use of Non-quadratic Properties of the Objective Function , 1971 .

[9]  Dong-Hui Li,et al.  Spectral Scaling BFGS Method , 2010 .

[10]  J. Nocedal,et al.  A tool for the analysis of Quasi-Newton methods with application to unconstrained minimization , 1989 .

[11]  D. Goldfarb.  A family of variable-metric methods derived by variational means , 1970 .

[12]  J. Borwein,et al.  Two-Point Step Size Gradient Methods , 1988 .

[13]  D. Shanno.  Conditioning of Quasi-Newton Methods for Function Minimization , 1970 .

[14]  Jorge Nocedal,et al.  On the Behavior of Broyden's Class of Quasi-Newton Methods , 1992, SIAM J. Optim..

[15]  M. Hestenes,et al.  Methods of conjugate gradients for solving linear systems , 1952 .

[16]  Walter F. Mascarenhas,et al.  The BFGS method with exact line searches fails for non-convex objective functions , 2004, Math. Program..

[17]  M. Fukushima,et al.  A modified BFGS method and its global convergence in nonconvex minimization , 2001 .

[18]  M. Powell.  On the Convergence of the Variable Metric Algorithm , 1971 .

[19]  Roger Fletcher,et al.  An Overview of Unconstrained Optimization , 1994 .

[20]  Jorge Nocedal,et al.  Theory of algorithms for unconstrained optimization , 1992, Acta Numerica.

[21]  D. Luenberger,et al.  SELF-SCALING VARIABLE METRIC ( SSVM ) ALGORITHMS Part I : Criteria and Sufficient Conditions for Scaling a Class of Algorithms * t , 2007 .

[22]  D. F. Shanno,et al.  Matrix conditioning and nonlinear optimization , 1978, Math. Program..

[23]  M. C. Biggs.  A Note on Minimization Algorithms which make Use of Non-quardratic Properties of the Objective Function , 1973 .

[24]  P. Wolfe.  Convergence Conditions for Ascent Methods. II: Some Corrections , 1971 .

[25]  Philip E. Gill,et al.  Reduced-Hessian Quasi-Newton Methods for Unconstrained Optimization , 2001, SIAM J. Optim..

[26]  H. Walker.  Quasi-Newton Methods , 1978 .

[27]  R. Kellogg.  A nonlinear alternating direction method , 1969 .

[28]  Dirk Siegel,et al.  Modifying the BFGS update by a new column scaling technique , 1994, Math. Program..

[29]  P. Wolfe.  Convergence Conditions for Ascent Methods. II , 1969 .

[30]  J. J. Moré,et al.  Quasi-Newton Methods, Motivation and Theory , 1974 .

[31]  Neculai Andrei,et al.  An adaptive scaled BFGS method for unconstrained optimization , 2018, Numerical Algorithms.

[32]  Gonglin Yuan,et al.  The Superlinear Convergence of a Modified BFGS-Type Method for Unconstrained Optimization , 2004, Comput. Optim. Appl..

[33]  Jorge Nocedal,et al.  Automatic Column Scaling Strategies for Quasi-Newton Methods , 1993, SIAM J. Optim..

[34]  Jianzhon Zhang,et al.  Properties and numerical performance of quasi-Newton methods with modified quasi-Newton equations , 2001 .

[35]  Nicholas I. M. Gould,et al.  CUTE: constrained and unconstrained testing environment , 1995, TOMS.

[36]  J. Nocedal,et al.  Global Convergence of a Class of Quasi-newton Methods on Convex Problems, Siam Some Global Convergence Properties of a Variable Metric Algorithm for Minimization without Exact Line Searches, Nonlinear Programming, Edited , 1996 .

[37]  Neculai Andrei,et al.  An Unconstrained Optimization Test Functions Collection , 2008 .

[38]  E. Polak.  Introduction to linear and nonlinear programming , 1973 .

[39]  C. G. Broyden.  The Convergence of a Class of Double-rank Minimization Algorithms 1. General Considerations , 1970 .

[40]  Gonglin Yuan,et al.  Convergence analysis of a modified BFGS method on convex minimizations , 2010, Comput. Optim. Appl..

[41]  Jorge J. Moré,et al.  Benchmarking optimization software with performance profiles , 2001, Math. Program..

[42]  Jorge Nocedal,et al.  Analysis of a self-scaling quasi-Newton method , 1993, Math. Program..

[43]  D. Luenberger,et al.  Self-Scaling Variable Metric (SSVM) Algorithms , 1974 .

[44]  L. Dixon.  Variable metric algorithms: Necessary and sufficient conditions for identical behavior of nonquadratic functions , 1972 .

[45]  Yu-Hong Dai,et al.  Convergence Properties of the BFGS Algoritm , 2002, SIAM J. Optim..

[46]  Ya-Xiang Yuan,et al.  Optimization theory and methods , 2006 .

[47]  Xiwen Lu,et al.  Global convergence of BFGS and PRP methods under a modified weak Wolfe–Powell line search , 2017 .

[48]  J. J. Moré,et al.  A Characterization of Superlinear Convergence and its Application to Quasi-Newton Methods , 1973 .

[49]  David F. Shanno,et al.  Conjugate Gradient Methods with Inexact Searches , 1978, Math. Oper. Res..

[50]  Neculai Andrei,et al.  Eigenvalues versus singular values study in conjugate gradient algorithms for large-scale unconstrained optimization , 2017, Optim. Methods Softw..

[51]  Mehiddin Al-Baali.  Global and Superlinear Convergence of a Restricted Class of Self-Scaling Methods with Inexact Line Searches, for Convex Functions , 1998, Comput. Optim. Appl..

[52]  R. Fletcher,et al.  A New Approach to Variable Metric Algorithms , 1970, Comput. J..

[53]  Siam Rfview,et al.  CONVERGENCE CONDITIONS FOR ASCENT METHODS , 2016 .