Adaptive cubic overestimation methods for unconstrained optimization. Part I: motivation, convergence and numerical results

An Adaptive Cubic Overestimation (ACO) algorithm for unconstrained optimization is proposed, generalizing at the same time an unpublished method due to Griewank (Technical Report NA/12, 1981, DAMTP, Univ. of Cambridge), an algorithm by Nesterov & Polyak (Math. Programming 108(1), 2006, pp 177-205) and a proposal by Weiser, Deuflhard & Erdmann (Optim. Methods Softw. 22(3), 2007, pp 413-431). At each iteration of our approach, an approximate global minimizer of a local cubic regularization of the objective function is determined, and this ensures a significant improvement in the objective so long as the Hessian of the objective is locally Lipschitz continuous. The new method uses an adaptive estimation of the local Lipschitz constant and approximations to the global model-minimizer which remain computationally-viable even for large-scale problems. We show that the excellent global and local convergence properties obtained by Nesterov & Polyak are retained, and sometimes extended to a wider class of problems, by our ACO approach. Numerical experiments with small-scale test problems from the CUTEr set show superior performance of the ACO algorithm when compared to a trust-region implementation.

[1]  J. J. Moré,et al.  A Characterization of Superlinear Convergence and its Application to Quasi-Newton Methods , 1973 .

[2]  S. W. Thomas.  Sequential estimation techniques for quasi-newton algorithms. , 1975 .

[3]  Jorge J. Moré,et al.  Recent Developments in Algorithms and Software for Trust Region Methods , 1982, ISMP.

[4]  R. Dembo,et al.  INEXACT NEWTON METHODS , 1982 .

[5]  Trond Steihaug,et al.  Truncated-newtono algorithms for large-scale unconstrained optimization , 1983, Math. Program..

[6]  Gene H. Golub,et al.  Matrix computations , 1983 .

[7]  John E. Dennis,et al.  Numerical methods for unconstrained optimization and nonlinear equations , 1983, Prentice Hall series in computational mathematics.

[8]  Andreas Griewank,et al.  Numerical experiments with partially separable optimization problems , 1984 .

[9]  Anderas Griewank.  The “global” convergence of Broyden-like methods with suitable line search , 1986, The Journal of the Australian Mathematical Society. Series B. Applied Mathematics.

[10]  M. Grigoriadis,et al.  A fast algorithm for Trummer's problem , 1987 .

[11]  A. Gerasoulis.  A fast algorithm for the multiplication of generalized Hilbert matrices with vectors , 1988 .

[12]  Nicholas I. M. Gould,et al.  Convergence of quasi-Newton matrices generated by the symmetric rank one update , 1991, Math. Program..

[13]  Richard H. Byrd,et al.  Analysis of a Symmetric Rank-One Trust Region Method , 1996, SIAM J. Optim..

[14]  Stephen J. Wright,et al.  Numerical Optimization , 2018, Fundamental Statistical Inference.

[15]  Nicholas I. M. Gould,et al.  Solving the Trust-Region Subproblem using the Lanczos Method , 1999, SIAM J. Optim..

[16]  Nicholas I. M. Gould,et al.  Trust Region Methods , 2000, MOS-SIAM Series on Optimization.

[17]  Jorge J. Moré,et al.  Digital Object Identifier (DOI) 10.1007/s101070100263 , 2001 .

[18]  Nicholas I. M. Gould,et al.  CUTEr and SifDec: A constrained and unconstrained testing environment, revisited , 2003, TOMS.

[19]  Peter Deuflhard,et al.  Newton Methods for Nonlinear Problems , 2004 .

[20]  Yurii Nesterov,et al.  Introductory Lectures on Convex Optimization - A Basic Course , 2014, Applied Optimization.

[21]  Nicholas I. M. Gould,et al.  Sensitivity of trust-region algorithms to their parameters , 2005, 4OR.

[22]  Yurii Nesterov,et al.  Cubic regularization of Newton method and its global performance , 2006, Math. Program..

[23]  P. Toint,et al.  Adaptive cubic overestimation methods for unconstrained optimization . Part II : worst-case iteration complexity , 2007 .

[24]  Peter Deuflhard,et al.  Affine conjugate adaptive Newton methods for nonlinear elastomechanics , 2007, Optim. Methods Softw..

[25]  P. Toint,et al.  A recursive trust-region method in infinity norm for bound-constrained nonlinear optimization , 2007 .

[26]  Serge Gratton,et al.  Recursive Trust-Region Methods for Multiscale Nonlinear Optimization , 2008, SIAM J. Optim..

[27]  Yurii Nesterov,et al.  Accelerating the cubic regularization of Newton’s method on convex problems , 2005, Math. Program..