Strong Evaluation Complexity Bounds for Arbitrary-Order Optimization of Nonconvex Nonsmooth Composite Functions

We introduce the concept of strong high-order approximate minimizers for nonconvex optimization problems. These apply in both standard smooth and composite non-smooth settings, and additionally allow convex or inexpensive constraints. An adaptive regularization algorithm is then proposed to find such approximate minimizers. Under suitable Lipschitz continuity assumptions, whenever the feasible set is convex, it is shown that using a model of degree $p$, this algorithm will find a strong approximate q-th-order minimizer in at most ${\cal O}\left(\max_{1\leq j\leq q}\epsilon_j^{-(p+1)/(p-j+1)}\right)$ evaluations of the problem's functions and their derivatives, where $\epsilon_j$ is the $j$-th order accuracy tolerance; this bound applies when either $q=1$ or the problem is not composite with $q \leq 2$. For general non-composite problems, even when the feasible set is nonconvex, the bound becomes ${\cal O}\left(\max_{1\leq j\leq q}\epsilon_j^{-q(p+1)/p}\right)$ evaluations. If the problem is composite, and either $q > 1$ or the feasible set is not convex, the bound is then ${\cal O}\left(\max_{1\leq j\leq q}\epsilon_j^{-(q+1)}\right)$ evaluations. These results not only provide, to our knowledge, the first known bound for (unconstrained or inexpensively-constrained) composite problems for optimality orders exceeding one, but also give the first sharp bounds for high-order strong approximate $q$-th order minimizers of standard (unconstrained and inexpensively constrained) smooth problems, thereby complementing known results for weak minimizers.

[1]  A. Bruckner,et al.  SOME FUNCTION CLASSES RELATED TO THE CLASS OF CONVEX FUNCTIONS , 1962 .

[2]  T. M. Williams,et al.  Practical Methods of Optimization. Vol. 1: Unconstrained Optimization , 1980 .

[3]  P. Hansen.  Rank-Deficient and Discrete Ill-Posed Problems: Numerical Aspects of Linear Inversion , 1987 .

[4]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[5]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[6]  Zvi Drezner,et al.  Facility location - applications and theory , 2001 .

[7]  Yurii Nesterov,et al.  Cubic regularization of Newton method and its global performance , 2006, Math. Program..

[8]  P. Toint,et al.  Adaptive cubic overestimation methods for unconstrained optimization , 2007 .

[9]  Marc Teboulle,et al.  A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , 2009, SIAM J. Imaging Sci..

[10]  Nicholas I. M. Gould,et al.  On the Evaluation Complexity of Composite Function Minimization with Applications to Nonconvex Nonlinear Programming , 2011, SIAM J. Optim..

[11]  P. Toint,et al.  Improved worst-case evaluation complexity for potentially rank-deficient nonlinear least-Euclidean-norm problems using higher-order regularized models , 2015 .

[12]  S. Frick,et al.  Compressed Sensing , 2014, Computer Vision, A Reference Guide.

[13]  Stephen J. Wright,et al.  A proximal method for composite minimization , 2008, Mathematical Programming.

[14]  José Mario Martínez,et al.  Worst-case evaluation complexity for unconstrained nonlinear optimization using high-order regularized models , 2017, Math. Program..

[15]  Daniel P. Robinson,et al.  An Inexact Regularized Newton Framework with a Worst-Case Iteration Complexity of $\mathcal{O}(\epsilon^{-3/2})$ for Nonconvex Optimization , 2017, 1708.00475.

[16]  Hong Wang,et al.  Partially separable convexly-constrained optimization with non-Lipschitzian singularities and its complexity , 2017, ArXiv.

[17]  Nicholas I. M. Gould,et al.  Worst-case evaluation complexity of regularization methods for smooth unconstrained optimization using Hölder continuous gradients , 2017, Optim. Methods Softw..

[18]  Stephen J. Wright,et al.  Complexity Analysis of Second-Order Line-Search Algorithms for Smooth Nonconvex Optimization , 2017, SIAM J. Optim..

[19]  Frank E. Curtis.  An inexact regularized Newton framework with a worst-case iteration complexity of O(ε−3/2) for nonconvex optimization , 2018 .

[20]  Nicholas I. M. Gould,et al.  Second-Order Optimality and Beyond: Characterization and Evaluation Complexity in Convexly Constrained Nonlinear Optimization , 2018, Found. Comput. Math..

[21]  Daniel Brand,et al.  Training Deep Neural Networks with 8-bit Floating Point Numbers , 2018, NeurIPS.

[22]  Xiaojun Chen,et al.  Complexity of Partially Separable Convexly Constrained Optimization with Non-Lipschitzian Singularities , 2019, SIAM J. Optim..

[23]  S. Bellavia,et al.  Adaptive Regularization Algorithms with Inexact Evaluations for Nonconvex Optimization , 2018, SIAM J. Optim..

[24]  Nicholas I. M. Gould,et al.  Sharp worst-case evaluation complexity bounds for arbitrary-order nonconvex optimization with inexpensive constraints , 2018, SIAM J. Optim..

[25]  X. Chen,et al.  High-order evaluation complexity for convexly-constrained optimization with non-Lipschitzian group sparsity terms , 2019, Math. Program..

[26]  E. Simon,et al.  An algorithm for the minimization of nonsmooth nonconvex functions using inexact evaluations and its worst-case complexity , 2019, Math. Program..