Adaptive regularization for nonconvex optimization using inexact function values and randomly perturbed derivatives

Abstract A regularization algorithm allowing random noise in derivatives and inexact function values is proposed for computing approximate local critical points of any order for smooth unconstrained optimization problems. For an objective function with Lipschitz continuous p-th derivative and given an arbitrary optimality order q ≤ p , an upper bound on the number of function and derivatives evaluations is established for this algorithm. This bound is in expectation, and in terms of a power of the required tolerances, this power depending on whether q ≤ 2 or q > 2 . Moreover these bounds are sharp in the order of the accuracy tolerances. An extension to convexly constrained problems is also outlined.

[1]  Etienne de Klerk,et al.  Worst-Case Examples for Lasserre's Measure-Based Hierarchy for Polynomial Optimization on the Hypercube , 2020, Math. Oper. Res..

[2]  Yurii Nesterov,et al.  Introductory Lectures on Convex Optimization - A Basic Course , 2014, Applied Optimization.

[3]  Daniel P. Robinson,et al.  Regional complexity analysis of algorithms for nonconvex smooth optimization , 2018, Mathematical Programming.

[4]  Yurii Nesterov,et al.  Cubic regularization of Newton method and its global performance , 2006, Math. Program..

[5]  Stephen A. Vavasis,et al.  Black-Box Complexity of Local Minimization , 1993, SIAM J. Optim..

[6]  E. Simon,et al.  An algorithm for the minimization of nonsmooth nonconvex functions using inexact evaluations and its worst-case complexity , 2019, Math. Program..

[7]  Xiaojun Chen,et al.  Complexity analysis of interior point algorithms for non-Lipschitz and nonconvex minimization , 2015, Math. Program..

[8]  Stefania Bellavia,et al.  Stochastic analysis of an adaptive cubic regularization method under inexact gradient evaluations and dynamic Hessian accuracy , 2020, Optimization.

[9]  K. Scheinberg,et al.  A Theoretical and Empirical Comparison of Gradient Approximations in Derivative-Free Optimization , 2019, Foundations of Computational Mathematics.

[10]  Katya Scheinberg,et al.  Stochastic optimization using a trust-region method and random models , 2015, Mathematical Programming.

[11]  Nicholas I. M. Gould,et al.  Adaptive cubic regularisation methods for unconstrained optimization. Part II: worst-case function- and derivative-evaluation complexity , 2011, Math. Program..

[12]  STEFANIA BELLAVIA,et al.  Adaptive cubic regularization methods with dynamic inexact Hessian information and applications to finite-sum minimization , 2018, IMA Journal of Numerical Analysis.

[13]  E. D. Klerk,et al.  Convergence analysis of a Lasserre hierarchy of upper bounds for polynomial minimization on the sphere , 2019, Mathematical programming.

[14]  M. Laurent,et al.  Improved convergence analysis of Lasserre’s measure-based upper bounds for polynomial minimization on compact sets , 2019, Mathematical programming.

[15]  S. Bellavia,et al.  Adaptive Regularization Algorithms with Inexact Evaluations for Nonconvex Optimization , 2018, SIAM J. Optim..

[16]  Yair Carmon,et al.  Gradient Descent Finds the Cubic-Regularized Nonconvex Newton Step , 2019, SIAM J. Optim..

[17]  Yurii Nesterov,et al.  Random Gradient-Free Minimization of Convex Functions , 2015, Foundations of Computational Mathematics.

[18]  Peng Xu,et al.  Newton-type methods for non-convex optimization under inexact Hessian information , 2017, Math. Program..

[19]  Katya Scheinberg,et al.  Convergence Rate Analysis of a Stochastic Trust-Region Method via Supermartingales , 2016, INFORMS Journal on Optimization.

[20]  Yair Carmon,et al.  Second-Order Information in Non-Convex Stochastic Optimization: Power and Limitations , 2020, COLT.

[21]  Nicholas I. M. Gould,et al.  On the Complexity of Steepest Descent, Newton's and Regularized Newton's Methods for Nonconvex Unconstrained Optimization Problems , 2010, SIAM J. Optim..

[22]  S. Vavasis.  Nonlinear optimization: complexity issues , 1991 .

[23]  Katya Scheinberg,et al.  Adaptive Stochastic Optimization: A Framework for Analyzing Stochastic Optimization Algorithms , 2020, IEEE Signal Processing Magazine.

[24]  Irina S. Dolinskaya,et al.  A Derivative-Free Trust-Region Algorithm for the Optimization of Functions Smoothed via Gaussian Convolution Using Adaptive Multiple Importance Sampling , 2018, SIAM J. Optim..

[25]  Nicholas I. M. Gould,et al.  Sharp worst-case evaluation complexity bounds for arbitrary-order nonconvex optimization with inexpensive constraints , 2018, SIAM J. Optim..

[26]  José Mario Martínez,et al.  Worst-case evaluation complexity for unconstrained nonlinear optimization using high-order regularized models , 2017, Math. Program..

[27]  Nicholas I. M. Gould,et al.  Trust Region Methods , 2000, MOS-SIAM Series on Optimization.

[28]  Stephen J. Wright,et al.  A Newton-CG algorithm with complexity guarantees for smooth unconstrained optimization , 2018, Mathematical Programming.

[29]  Matthias Heinkenschloss,et al.  Inexact Objective Function Evaluations in a Trust-Region Algorithm for PDE-Constrained Optimization under Uncertainty , 2014, SIAM J. Sci. Comput..

[30]  Katya Scheinberg,et al.  Global convergence rate analysis of unconstrained optimization methods based on probabilistic models , 2015, Mathematical Programming.

[31]  P. Toint,et al.  An adaptive cubic regularization algorithm for nonconvex optimization with convex constraints and its function-evaluation complexity , 2012 .

[32]  Chen Ling,et al.  The Best Rank-1 Approximation of a Symmetric Tensor and Related Spherical Optimization Problems , 2012, SIAM J. Matrix Anal. Appl..

[33]  Katya Scheinberg,et al.  A Stochastic Line Search Method with Expected Complexity Analysis , 2020, SIAM J. Optim..