Cubic regularization in symmetric rank-1 quasi-Newton methods

Quasi-Newton methods based on the symmetric rank-one (SR1) update have been known to be fast and provide better approximations of the true Hessian than popular rank-two approaches, but these properties are guaranteed under certain conditions which frequently do not hold. Additionally, SR1 is plagued by the lack of guarantee of positive definiteness for the Hessian estimate. In this paper, we propose cubic regularization as a remedy to relax the conditions on the proofs of convergence for both speed and accuracy and to provide a positive definite approximation at each step. We show that the n-step convergence property for strictly convex quadratic programs is retained by the proposed approach. Extensive numerical results on unconstrained problems from the CUTEr test set are provided to demonstrate the computational efficiency and robustness of the approach.

[1]  A. Schiela.  A Flexible Framework for Cubic Regularization Algorithms for Nonconvex Optimization in Function Space , 2019, Numerical Functional Analysis and Optimization.

[2]  Jorge J. Moré,et al.  Digital Object Identifier (DOI) 10.1007/s101070100263 , 2001 .

[3]  David F. Shanno,et al.  Conjugate Gradient Methods with Inexact Searches , 1978, Math. Oper. Res..

[4]  Marco Sciandrone,et al.  On the use of iterative methods in cubic regularization for unconstrained optimization , 2015, Comput. Optim. Appl..

[5]  Nicholas I. M. Gould,et al.  Adaptive cubic regularisation methods for unconstrained optimization. Part I: motivation, convergence and numerical results , 2011, Math. Program..

[6]  David F. Shanno,et al.  Interior-point methods for nonconvex nonlinear programming: cubic regularization , 2014, Comput. Optim. Appl..

[7]  Richard H. Byrd,et al.  Analysis of a Symmetric Rank-One Trust Region Method , 1996, SIAM J. Optim..

[8]  Anthony V. Fiacco,et al.  Nonlinear programming;: Sequential unconstrained minimization techniques , 1968 .

[9]  William C. Davidon,et al.  Variable Metric Method for Minimization , 1959, SIAM J. Optim..

[10]  C. G. Broyden.  The Convergence of a Class of Double-rank Minimization Algorithms 2. The New Algorithm , 1970 .

[11]  S. Ulbrich,et al.  Geometry Optimization of Branched Sheet Metal Products , 2012 .

[12]  R. Fletcher,et al.  A New Approach to Variable Metric Algorithms , 1970, Comput. J..

[13]  Anima Anandkumar,et al.  Efficient approaches for escaping higher order saddle points in non-convex optimization , 2016, COLT.

[14]  D. Goldfarb.  A family of variable-metric methods derived by variational means , 1970 .

[15]  Stefania Bellavia,et al.  Strong local convergence properties of adaptive regularized methods for nonlinear least squares , 2015 .

[16]  M. J. D. Powell,et al.  Recent advances in unconstrained optimization , 1971, Math. Program..

[17]  C. G. Broyden.  Quasi-Newton methods and their application to function minimisation , 1967 .

[18]  Stephen J. Wright,et al.  Numerical Optimization (Springer Series in Operations Research and Financial Engineering) , 2000 .

[19]  William C. Davidon,et al.  Variance Algorithm for Minimization , 1968, Comput. J..

[20]  Yurii Nesterov,et al.  Cubic regularization of Newton method and its global performance , 2006, Math. Program..

[21]  S. Oren.  SELF-SCALING VARIABLE METRIC (SSVM) ALGORITHMS Part II: Implementation and Experiments*t , 1974 .

[22]  Robert J. Vanderbei,et al.  An Interior-Point Algorithm for Nonconvex Nonlinear Programming , 1999, Comput. Optim. Appl..

[23]  P. Wolfe.  Convergence Conditions for Ascent Methods. II , 1969 .

[24]  Richard H. Byrd,et al.  A Theoretical and Experimental Study of the Symmetric Rank-One Update , 1993, SIAM J. Optim..

[25]  Roger Fletcher,et al.  A Rapidly Convergent Descent Method for Minimization , 1963, Comput. J..

[26]  P. Wolfe.  Convergence Conditions for Ascent Methods. II: Some Corrections , 1971 .

[27]  José Mario Martínez,et al.  Worst-case evaluation complexity for unconstrained nonlinear optimization using high-order regularized models , 2017, Math. Program..

[28]  高自友,et al.  关于“SUFFICIENT CONDITIONS FOR THE CONVERGENCE OF A VAR … , 1990 .

[29]  Nicholas I. M. Gould,et al.  Updating the regularization parameter in the adaptive cubic regularization algorithm , 2012, Comput. Optim. Appl..

[30]  José Mario Martínez,et al.  Cubic-regularization counterpart of a variable-norm trust-region method for unconstrained minimization , 2017, J. Glob. Optim..

[31]  T. Cullen.  Global , 1981 .

[32]  S. I. Feldman,et al.  A Fortran to C converter , 1990, FORF.

[33]  Lue Li,et al.  A trust region algorithm with adaptive cubic regularization methods for nonsmooth convex minimization , 2012, Comput. Optim. Appl..

[34]  J. Dussault.  Simple unified convergence proofs for Trust Region and a new ARC variant , 2015 .

[35]  D. Luenberger,et al.  Self-Scaling Variable Metric (SSVM) Algorithms , 1974 .

[36]  Shmuel S. Oren,et al.  Optimal conditioning of self-scaling variable Metric algorithms , 1976, Math. Program..

[37]  H. Y. Huang.  Unified approach to quadratically convergent algorithms for function minimization , 1970 .

[38]  Marco Sciandrone,et al.  A cubic regularization algorithm for unconstrained optimization using line search and nonmonotone techniques , 2016, Optim. Methods Softw..

[39]  Nicholas I. M. Gould,et al.  Convergence of quasi-Newton matrices generated by the symmetric rank one update , 1991, Math. Program..

[40]  Brian W. Kernighan,et al.  AMPL: A Modeling Language for Mathematical Programming , 1993 .

[41]  D. Luenberger,et al.  SELF-SCALING VARIABLE METRIC ( SSVM ) ALGORITHMS Part I : Criteria and Sufficient Conditions for Scaling a Class of Algorithms * t , 2007 .

[42]  D. F. Shanno,et al.  Matrix conditioning and nonlinear optimization , 1978, Math. Program..

[43]  Charles B. Dunham,et al.  Remark on “Algorithm 500: Minimization of Unconstrained Multivariate Functions [E4]” , 1977, TOMS.

[44]  Ruey-Lin Sheu,et al.  On the p-regularized trust region subproblem , 2014 .

[45]  R. Fletcher.  Practical Methods of Optimization , 1988 .

[46]  D. Shanno.  Conditioning of Quasi-Newton Methods for Function Minimization , 1970 .

[47]  Nicholas I. M. Gould,et al.  Adaptive cubic regularisation methods for unconstrained optimization. Part II: worst-case function- and derivative-evaluation complexity , 2011, Math. Program..

[48]  Peter Deuflhard,et al.  Affine conjugate adaptive Newton methods for nonlinear elastomechanics , 2007, Optim. Methods Softw..

[49]  Todd Munson,et al.  Benchmarking optimization software with COPS. , 2001 .

[50]  Andreas Griewank,et al.  Cubic overestimation and secant updating for unconstrained optimization of C2, 1 functions , 2014, Optim. Methods Softw..