Optimality of orders one to three and beyond: characterization and evaluation complexity in constrained nonconvex optimization

Necessary conditions for high-order optimality in smooth nonlinear constrained optimization are explored and their inherent intricacy discussed. A two-phase minimization algorithm is proposed which can achieve approximate first-, second- and third-order criticality and its evaluation complexity is analyzed as a function of the choice (among existing methods) of an inner algorithm for solving subproblems in each of the two phases. The relation between high-order criticality and penalization techniques is finally considered, showing that standard algorithmic approaches will fail if approximate constrained high-order critical points are sought.

[1]  Daniel P. Robinson,et al.  A trust region algorithm with a worst-case iteration complexity of O(ϵ-3/2)\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{docume , 2016, Mathematical Programming.

[2]  Nicholas I. M. Gould,et al.  Evaluation complexity bounds for smooth constrained nonlinear optimisation using scaled KKT conditions, high-order models and the criticality measure $χ$ , 2017, ArXiv.

[3]  Yurii Nesterov,et al.  Cubic regularization of Newton method and its global performance , 2006, Math. Program..

[4]  Nicholas I. M. Gould,et al.  Second-Order Optimality and Beyond: Characterization and Evaluation Complexity in Convexly Constrained Nonlinear Optimization , 2018, Found. Comput. Math..

[5]  José Mario Martínez,et al.  On High-order Model Regularization for Constrained Optimization , 2017, SIAM J. Optim..

[6]  H. Hancock.  Theory of Maxima and Minima , 1919 .

[7]  Nicholas I. M. Gould,et al.  On the Evaluation Complexity of Constrained Nonlinear Least-Squares and General Constrained Nonlinear Optimization Using Second-Order Methods , 2015, SIAM J. Numer. Anal..

[8]  Nicholas I. M. Gould,et al.  On the complexity of finding first-order critical points in constrained nonlinear optimization , 2014, Math. Program..

[9]  Katya Scheinberg,et al.  Complexity of Inexact Proximal Newton methods , 2013, ArXiv.

[10]  Hidefumi Kawaski,et al.  An envelope-like effect of infinitely many inequality constraints on second-order necessary conditions for minimization problems , 1988 .

[11]  José Mario Martínez,et al.  Worst-case evaluation complexity for unconstrained nonlinear optimization using high-order regularized models , 2017, Math. Program..

[12]  Jin Yun Yuan,et al.  Nonlinear Stepsize Control Algorithms: Complexity Bounds for First- and Second-Order Optimality , 2016, J. Optim. Theory Appl..

[13]  Nicholas I. M. Gould,et al.  Trust Region Methods , 2000, MOS-SIAM Series on Optimization.

[14]  P. Toint,et al.  Adaptive cubic overestimation methods for unconstrained optimization , 2007 .

[15]  Saeed Ghadimi,et al.  Accelerated gradient methods for nonconvex nonlinear and stochastic programming , 2013, Mathematical Programming.

[16]  Alexander Shapiro,et al.  Second Order Optimality Conditions Based on Parabolic Second Order Tangent Sets , 1999, SIAM J. Optim..

[17]  Anima Anandkumar,et al.  Efficient approaches for escaping higher order saddle points in non-convex optimization , 2016, COLT.

[18]  P. Toint,et al.  An adaptive cubic regularization algorithm for nonconvex optimization with convex constraints and its function-evaluation complexity , 2012 .

[19]  Nicholas I. M. Gould,et al.  On the Evaluation Complexity of Cubic Regularization Methods for Potentially Rank-Deficient Nonlinear Least-Squares Problems and Its Relevance to Constrained Nonlinear Optimization , 2013, SIAM J. Optim..

[20]  Hong Wang,et al.  Partially separable convexly-constrained optimization with non-Lipschitzian singularities and its complexity , 2017, ArXiv.

[21]  Nicholas I. M. Gould,et al.  On the Complexity of Steepest Descent, Newton's and Regularized Newton's Methods for Nonconvex Unconstrained Optimization Problems , 2010, SIAM J. Optim..

[22]  Katya Scheinberg,et al.  Practical inexact proximal quasi-Newton method with global complexity analysis , 2013, Mathematical Programming.

[23]  Xiaojun Chen,et al.  Complexity analysis of interior point algorithms for non-Lipschitz and nonconvex minimization , 2015, Math. Program..

[24]  Nicholas I. M. Gould,et al.  Adaptive cubic regularisation methods for unconstrained optimization. Part I: motivation, convergence and numerical results , 2011, Math. Program..

[25]  Hidefumi Kawasaki.  An envelope-like effect of infinitely many inequality constraints on second-order necessary conditions for minimization problems , 1988, Math. Program..

[26]  Daniel P. Robinson,et al.  Complexity Analysis of a Trust Funnel Algorithm for Equality Constrained Optimization , 2017, SIAM J. Optim..

[27]  N. Yamashita,et al.  Convergence Properties of the Regularized Newton Method for the Unconstrained Nonconvex Optimization , 2010 .

[28]  Nobuo Yamashita,et al.  On a Global Complexity Bound of the Levenberg-Marquardt Method , 2010, J. Optim. Theory Appl..

[29]  Serge Gratton,et al.  Recursive Trust-Region Methods for Multiscale Nonlinear Optimization , 2008, SIAM J. Optim..

[30]  P. Absil,et al.  Erratum to: ``Global rates of convergence for nonconvex optimization on manifolds'' , 2016, IMA Journal of Numerical Analysis.

[31]  Xiaojun Chen,et al.  Linearly Constrained Non-Lipschitz Optimization for Image Restoration , 2015, SIAM J. Imaging Sci..

[32]  Luís Nunes Vicente,et al.  Worst case complexity of direct search , 2013, EURO J. Comput. Optim..

[33]  José Mario Martínez,et al.  Cubic-regularization counterpart of a variable-norm trust-region method for unconstrained minimization , 2017, J. Glob. Optim..

[34]  Alexey Tret'yakov,et al.  Optimality Conditions for Degenerate Extremum Problems with Equality Constraints , 2003, SIAM J. Control. Optim..

[35]  Yurii Nesterov,et al.  Regularized Newton Methods for Minimizing Functions with Hölder Continuous Hessians , 2017, SIAM J. Optim..

[36]  P. Toint,et al.  Evaluation Complexity Bounds for Smooth Constrained Nonlinear Optimization Using Scaled KKT Conditions and High-Order Models , 2019, Approximation and Optimization.

[37]  O. Brezhneva,et al.  The pth-order optimality conditions for inequality constrained optimization problems , 2005 .

[38]  W. Hogan.  Point-to-Set Maps in Mathematical Programming , 1973 .

[39]  S. Łojasiewicz.  Ensembles semi-analytiques , 1965 .

[40]  Yinyu Ye,et al.  A note on the complexity of Lp minimization , 2011, Math. Program..

[41]  Serge Gratton,et al.  Direct Search Based on Probabilistic Descent , 2015, SIAM J. Optim..

[42]  Y. Nesterov,et al.  Globally Convergent Second-order Schemes for Minimizing Twice-differentiable Functions , 2016 .

[43]  Katya Scheinberg,et al.  Global convergence rate analysis of unconstrained optimization methods based on probabilistic models , 2015, Mathematical Programming.

[44]  Luís Nunes Vicente,et al.  Trust-Region Methods Without Using Derivatives: Worst Case Complexity and the NonSmooth Case , 2016, SIAM J. Optim..

[45]  José Mario Martínez,et al.  Evaluation Complexity for Nonlinear Constrained Optimization Using Unscaled KKT Conditions and High-Order Models , 2016, SIAM J. Optim..

[46]  J. Hiriart-Urruty,et al.  Convex analysis and minimization algorithms , 1993 .

[47]  Luís N. Vicente,et al.  On the optimal order of worst case complexity of direct search , 2016, Optim. Lett..

[48]  Lue Li,et al.  A trust region algorithm with adaptive cubic regularization methods for nonsmooth convex minimization , 2012, Comput. Optim. Appl..

[49]  J. Dussault.  Simple unified convergence proofs for Trust Region and a new ARC variant , 2015 .

[50]  P. Toint,et al.  Improved worst-case evaluation complexity for potentially rank-deficient nonlinear least-Euclidean-norm problems using higher-order regularized models , 2015 .

[51]  Serge Gratton,et al.  A decoupled first/second-order steps technique for nonconvex nonlinear unconstrained optimization with improved complexity bounds , 2020, Math. Program..

[52]  P. Toint,et al.  Worst-case evaluation complexity of non-monotone gradient-related algorithms for unconstrained optimization , 2015 .

[53]  P. Toint,et al.  Adaptive cubic overestimation methods for unconstrained optimization. Part I: motivation, convergence and numerical results , 2008 .

[54]  Florian Jarre,et al.  On Nesterov's smooth Chebyshev–Rosenbrock function , 2013, Optim. Methods Softw..

[55]  Ya-xiang Yuan,et al.  On the convergence and worst-case complexity of trust-region and regularization methods for unconstrained optimization , 2015, Math. Program..

[56]  Jean-Pierre Dussault.  ARCq: a new adaptive regularization by cubics , 2018, Optim. Methods Softw..

[57]  Xiaojun Chen,et al.  Worst-Case Complexity of Smoothing Quadratic Regularization Methods for Non-Lipschitzian Optimization , 2013, SIAM J. Optim..

[58]  Serge Gratton,et al.  On the use of the energy norm in trust-region and adaptive cubic regularization subproblems , 2017, Comput. Optim. Appl..

[59]  R. Cominetti.  Metric regularity, tangent sets, and second-order optimality conditions , 1990 .