An algorithm for the minimization of nonsmooth nonconvex functions using inexact evaluations and its worst-case complexity

An adaptive regularization algorithm using inexact function and derivatives evaluations is proposed for the solution of composite nonsmooth nonconvex optimization. It is shown that this algorithm needs at most $$O(|\log (\epsilon )|\,\epsilon ^{-2})$$ O ( | log ( ϵ ) | ϵ - 2 ) evaluations of the problem’s functions and their derivatives for finding an $$\epsilon $$ ϵ -approximate first-order stationary point. This complexity bound therefore generalizes that provided by Bellavia et al. (Theoretical study of an adaptive cubic regularization method with dynamic inexact Hessian information. arXiv:1808.06239 , 2018) for inexact methods for smooth nonconvex problems, and is within a factor $$|\log (\epsilon )|$$ | log ( ϵ ) | of the optimal bound known for smooth and nonsmooth nonconvex minimization with exact evaluations. A practically more restrictive variant of the algorithm with worst-case complexity $$O(|\log (\epsilon )|+\epsilon ^{-2})$$ O ( | log ( ϵ ) | + ϵ - 2 ) is also presented.

[1]  Vyacheslav Kungurtsev,et al.  A Subsampling Line-Search Method with Second-Order Results , 2018, INFORMS J. Optim..

[2]  Stefania Bellavia,et al.  Deterministic and stochastic inexact regularization algorithms for nonconvex optimization with optimal complexity , 2018, ArXiv.

[3]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[4]  Nicholas I. M. Gould,et al.  Trust Region Methods , 2000, MOS-SIAM Series on Optimization.

[5]  Dmitriy Drusvyatskiy,et al.  Stochastic model-based minimization of weakly convex functions , 2018, SIAM J. Optim..

[6]  David L Donoho,et al.  Compressed sensing , 2006, IEEE Transactions on Information Theory.

[7]  Serge Gratton,et al.  Recursive Trust-Region Methods for Multiscale Nonlinear Optimization , 2008, SIAM J. Optim..

[8]  Nicholas I. M. Gould,et al.  On the Oracle Complexity of First-Order and Derivative-Free Algorithms for Smooth Nonconvex Minimization , 2012, SIAM J. Optim..

[9]  Katya Scheinberg,et al.  Convergence Rate Analysis of a Stochastic Trust-Region Method via Supermartingales , 2016, INFORMS Journal on Optimization.

[10]  Nicholas I. M. Gould,et al.  On the Evaluation Complexity of Composite Function Minimization with Applications to Nonconvex Nonlinear Programming , 2011, SIAM J. Optim..

[11]  Warren Hare,et al.  A proximal bundle method for nonsmooth nonconvex functions with inexact information , 2015, Computational Optimization and Applications.

[12]  D. L. Donoho,et al.  Compressed sensing , 2006, IEEE Trans. Inf. Theory.

[13]  P. Hansen.  Rank-Deficient and Discrete Ill-Posed Problems: Numerical Aspects of Linear Inversion , 1987 .

[14]  Nicholas I. M. Gould,et al.  Sharp worst-case evaluation complexity bounds for arbitrary-order nonconvex optimization with inexpensive constraints , 2018, SIAM J. Optim..

[15]  José Mario Martínez,et al.  Worst-case evaluation complexity for unconstrained nonlinear optimization using high-order regularized models , 2017, Math. Program..

[16]  Tianyi Lin,et al.  On Adaptive Cubic Regularized Newton's Methods for Convex Optimization via Random Sampling , 2018 .

[17]  Marc Teboulle,et al.  A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , 2009, SIAM J. Imaging Sci..

[18]  Cho-Jui Hsieh,et al.  Stochastic Second-order Methods for Non-convex Optimization with Inexact Hessian and Gradient , 2018, ArXiv.

[19]  Stephen J. Wright,et al.  A proximal method for composite minimization , 2008, Mathematical Programming.

[20]  Katya Scheinberg,et al.  Global convergence rate analysis of unconstrained optimization methods based on probabilistic models , 2015, Mathematical Programming.

[21]  Feng Ruan,et al.  Stochastic Methods for Composite and Weakly Convex Optimization Problems , 2017, SIAM J. Optim..

[22]  Richard G. Carter,et al.  Numerical Experience with a Class of Algorithms for Nonlinear Optimization Using Inexact Function and Gradient Information , 1993, SIAM J. Sci. Comput..

[23]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[24]  Ya-Xiang Yuan,et al.  Conditions for convergence of trust region algorithms for nonsmooth optimization , 1985, Math. Program..

[25]  Daniel Brand,et al.  Training Deep Neural Networks with 8-bit Floating Point Numbers , 2018, NeurIPS.

[26]  Tianyi Lin,et al.  Adaptively Accelerating Cubic Regularized Newton's Methods for Convex Optimization via Random Sampling , 2018 .

[27]  Serge Gratton,et al.  A note on solving nonlinear optimization problems in variable precision , 2018, Computational Optimization and Applications.

[28]  Alexander J. Smola,et al.  Proximal Stochastic Methods for Nonsmooth Nonconvex Finite-Sum Optimization , 2016, NIPS.

[29]  Matthias Heinkenschloss,et al.  Inexact Objective Function Evaluations in a Trust-Region Algorithm for PDE-Constrained Optimization under Uncertainty , 2014, SIAM J. Sci. Comput..

[30]  S. Bellavia,et al.  Adaptive Regularization Algorithms with Inexact Evaluations for Nonconvex Optimization , 2018, SIAM J. Optim..

[31]  Peng Xu,et al.  Newton-type methods for non-convex optimization under inexact Hessian information , 2017, Math. Program..

[32]  Michael W. Mahoney,et al.  Sub-sampled Newton methods , 2018, Math. Program..

[33]  Yurii Nesterov,et al.  Introductory Lectures on Convex Optimization - A Basic Course , 2014, Applied Optimization.

[34]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.