Universal regularization methods - varying the power, the smoothness and the accuracy

Adaptive cubic regularization methods have emerged as a credible alternative to linesearch and trust-region for smooth nonconvex optimization, with optimal complexity amongst second-order methods. Here we consider a general/new class of adaptive regularization methods, that use first- or higher-order local Taylor models of the objective regularized by a(ny) power of the step size and applied to convexly-constrained optimization problems. We investigate the worst-case evaluation complexity/global rate of convergence of these algorithms, when the level of sufficient smoothness of the objective may be unknown or may even be absent. We find that the methods accurately reflect in their complexity the degree of smoothness of the objective and satisfy increasingly better bounds with improving accuracy of the models. The bounds vary continuously and robustly with respect to the regularization power and accuracy of the model and the degree of smoothness of the objective.

[1]  Y. Nesterov,et al.  Globally Convergent Second-order Schemes for Minimizing Twice-differentiable Functions , 2016 .

[2]  K. Schittkowski,et al.  NONLINEAR PROGRAMMING , 2022 .

[3]  Yurii Nesterov,et al.  Universal gradient methods for convex optimization problems , 2015, Math. Program..

[4]  P. Toint,et al.  Adaptive cubic overestimation methods for unconstrained optimization , 2007 .

[5]  Bobby Schnabel,et al.  Tensor Methods for Unconstrained Optimization Using Second Derivatives , 1991, SIAM J. Optim..

[6]  Nicholas I. M. Gould,et al.  Trust Region Methods , 2000, MOS-SIAM Series on Optimization.

[7]  Nicholas I. M. Gould,et al.  On the Complexity of Steepest Descent, Newton's and Regularized Newton's Methods for Nonconvex Unconstrained Optimization Problems , 2010, SIAM J. Optim..

[8]  Yurii Nesterov,et al.  Introductory Lectures on Convex Optimization - A Basic Course , 2014, Applied Optimization.

[9]  Yurii Nesterov,et al.  Implementable tensor methods in unconstrained convex optimization , 2019, Mathematical Programming.

[10]  John Darzentas,et al.  Problem Complexity and Method Efficiency in Optimization , 1983 .

[11]  José Mario Martínez,et al.  Worst-case evaluation complexity for unconstrained nonlinear optimization using high-order regularized models , 2017, Math. Program..

[12]  Yair Carmon,et al.  Lower bounds for finding stationary points I , 2017, Mathematical Programming.

[13]  Alain Bensoussan,et al.  Regularity Results for Nonlinear Elliptic Systems and Applications , 2002 .

[14]  J. Scott,et al.  A higher order method for solving nonlinear least-squares problems , 2017 .

[15]  Hong Wang,et al.  Partially separable convexly-constrained optimization with non-Lipschitzian singularities and its complexity , 2017, ArXiv.

[16]  P. Toint,et al.  Optimal Newton-type methods for nonconvex smooth optimization problems , 2011 .

[17]  José Mario Martínez,et al.  The Use of Quadratic Regularization with a Cubic Descent Condition for Unconstrained Optimization , 2017, SIAM J. Optim..

[18]  Nicholas I. M. Gould,et al.  Worst-case evaluation complexity of regularization methods for smooth unconstrained optimization using Hölder continuous gradients , 2017, Optim. Methods Softw..

[19]  Nicholas I. M. Gould,et al.  Convergence and evaluation-complexity analysis of a regularized tensor-Newton method for solving nonlinear least-squares problems , 2019, Comput. Optim. Appl..

[20]  Daniel P. Robinson,et al.  A trust region algorithm with a worst-case iteration complexity of O(ϵ-3/2)\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{docume , 2016, Mathematical Programming.

[21]  José Mario Martínez,et al.  On the use of third-order models with fourth-order regularization for unconstrained optimization , 2019, Optimization Letters.

[22]  Peter Deuflhard,et al.  Affine conjugate adaptive Newton methods for nonlinear elastomechanics , 2007, Optim. Methods Softw..

[23]  Yurii Nesterov,et al.  Cubic regularization of Newton method and its global performance , 2006, Math. Program..

[24]  Olivier Devolder,et al.  Exactness, inexactness and stochasticity in first-order methods for large-scale convex optimization , 2013 .