On Regularization and Active-set Methods with Complexity for Constrained Optimization

The main objective of this research is to introduce a practical method for smooth bound-constrained optimization that possesses worst-case evaluation complexity $O(\varepsilon^{-3/2})$ for finding an $\varepsilon$-approximate first-order stationary point when the Hessian of the objective function is Lipschitz continuous. As other well-established algorithms for optimization with box constraints, the algorithm proceeds visiting the different faces of the domain aiming to reduce the norm of an internal projected gradient and abandoning active constraints when no additional progress is expected in the current face. The introduced method emerges as a particular case of a method for minimization with linear constraints. Moreover, the linearly constrained minimization algorithm is an instance of a minimization algorithm with general constraints whose implementation may be unaffordable when the constraints are complicated. As a procedure for leaving faces, a different method is employed that may be regarded as a...

[1]  D K Smith,et al.  Numerical Optimization , 2001, J. Oper. Res. Soc..

[2]  Marco Sciandrone,et al.  A cubic regularization algorithm for unconstrained optimization using line search and nonmonotone techniques , 2016, Optim. Methods Softw..

[3]  Y. Nesterov,et al.  Globally Convergent Second-order Schemes for Minimizing Twice-differentiable Functions , 2016 .

[4]  J. M. Martínez,et al.  On sequential optimality conditions for smooth constrained optimization , 2011 .

[5]  José Mario Martínez,et al.  Spectral Projected Gradient Methods: Review and Perspectives , 2014 .

[6]  Jorge Nocedal,et al.  Remark on “algorithm 778: L-BFGS-B: Fortran subroutines for large-scale bound constrained optimization” , 2011, TOMS.

[7]  Benar Fux Svaiter,et al.  A Practical Optimality Condition Without Constraint Qualifications for Nonlinear Programming , 2003 .

[8]  José Mario Martínez,et al.  Cubic-regularization counterpart of a variable-norm trust-region method for unconstrained minimization , 2017, J. Glob. Optim..

[9]  Ernesto G. Birgin,et al.  2 . 2 Meaning of “ to solve a problem ” , 2011 .

[10]  Marco Sciandrone,et al.  On the use of iterative methods in cubic regularization for unconstrained optimization , 2015, Comput. Optim. Appl..

[11]  J. Dunn.  On the convergence of projected gradient processes to singular critical points , 1987 .

[12]  José Mario Martínez,et al.  Nonmonotone Spectral Projected Gradient Methods on Convex Sets , 1999, SIAM J. Optim..

[13]  José Mario Martínez,et al.  The Use of Quadratic Regularization with a Cubic Descent Condition for Unconstrained Optimization , 2017, SIAM J. Optim..

[14]  Michael A. Saunders,et al.  Large-scale linearly constrained optimization , 1978, Math. Program..

[15]  Ana Friedlander,et al.  On the numerical solution of bound constrained optimization problems , 1989 .

[16]  Yurii Nesterov,et al.  Regularized Newton Methods for Minimizing Functions with Hölder Continuous Hessians , 2017, SIAM J. Optim..

[17]  P. Toint,et al.  Evaluation Complexity Bounds for Smooth Constrained Nonlinear Optimization Using Scaled KKT Conditions and High-Order Models , 2019, Approximation and Optimization.

[18]  José Mario Martínez,et al.  Worst-case evaluation complexity for unconstrained nonlinear optimization using high-order regularized models , 2017, Math. Program..

[19]  Nicholas I. M. Gould,et al.  Complexity bounds for second-order optimality in unconstrained optimization , 2012, J. Complex..

[20]  P. Toint,et al.  Global convergence of a class of trust region algorithms for optimization with simple bounds , 1988 .

[21]  Zdenek Dostál,et al.  Projector preconditioning for partially bound‐constrained quadratic optimization , 2007, Numer. Linear Algebra Appl..

[22]  J. M. Martínez,et al.  Inexact spectral projected gradient methods on convex sets , 2003 .

[23]  J. M. Martínez,et al.  Practical active-set Euclidian trust-region method with spectral projected gradients for bound-constrained minimization , 2005 .

[24]  Yurii Nesterov,et al.  Cubic regularization of Newton method and its global performance , 2006, Math. Program..

[25]  José Mario Martínez,et al.  Evaluation Complexity for Nonlinear Constrained Optimization Using Unscaled KKT Conditions and High-Order Models , 2016, SIAM J. Optim..

[26]  José Mario Martínez,et al.  Partial spectral projected gradient method with active-set strategy for linearly constrained optimization , 2009, Numerical Algorithms.

[27]  Stephen J. Wright,et al.  Numerical Optimization (Springer Series in Operations Research and Financial Engineering) , 2000 .

[28]  R. Fletcher.  Practical Methods of Optimization , 1988 .

[29]  Frank E. Curtis.  An inexact regularized Newton framework with a worst-case iteration complexity of O(ε−3/2) for nonconvex optimization , 2018 .

[30]  Peter Deuflhard,et al.  Affine conjugate adaptive Newton methods for nonlinear elastomechanics , 2007, Optim. Methods Softw..

[31]  W. Marsden.  I and J , 2012 .

[32]  Nicholas I. M. Gould,et al.  Adaptive cubic regularisation methods for unconstrained optimization. Part II: worst-case function- and derivative-evaluation complexity , 2011, Math. Program..

[33]  José Mario Martínez,et al.  Large-Scale Active-Set Box-Constrained Optimization Method with Spectral Projected Gradients , 2002, Comput. Optim. Appl..

[34]  Dimitri P. Bertsekas,et al.  Nonlinear Programming , 1997 .

[35]  J. Dussault.  Simple unified convergence proofs for Trust Region and a new ARC variant , 2015 .

[36]  José Mario Martínez,et al.  PACKMOL: A package for building initial configurations for molecular dynamics simulations , 2009, J. Comput. Chem..

[37]  Nicholas I. M. Gould,et al.  On the Complexity of Steepest Descent, Newton's and Regularized Newton's Methods for Nonconvex Unconstrained Optimization Problems , 2010, SIAM J. Optim..

[38]  José Mario Martínez,et al.  Algorithm 813: SPG—Software for Convex-Constrained Optimization , 2001, TOMS.

[39]  Daniel P. Robinson,et al.  A trust region algorithm with a worst-case iteration complexity of O(ϵ-3/2)\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{docume , 2016, Mathematical Programming.

[40]  José Mario Martínez,et al.  On High-order Model Regularization for Constrained Optimization , 2017, SIAM J. Optim..

[41]  J. M. Martínez,et al.  Spectral projected gradient and variable metric methods for optimization with linear inequalities , 2005 .

[42]  Nicholas I. M. Gould,et al.  CUTEst: a Constrained and Unconstrained Testing Environment with safe threads for mathematical optimization , 2013, Computational Optimization and Applications.

[43]  Zdenek Dostál,et al.  Box Constrained Quadratic Programming with Proportioning and Projections , 1997, SIAM J. Optim..

[44]  J. Dunn.  Global and Asymptotic Convergence Rate Estimates for a Class of Projected Gradient Processes , 1981 .

[45]  J. Dunn.  Newton’s Method and the Goldstein Step-Length Rule for Constrained Minimization Problems , 1980 .

[46]  Nicholas I. M. Gould,et al.  Universal regularization methods - varying the power, the smoothness and the accuracy , 2018, 1811.07057.

[47]  D. Gleich.  TRUST REGION METHODS , 2017 .

[48]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[49]  P. Toint,et al.  An adaptive cubic regularization algorithm for nonconvex optimization with convex constraints and its function-evaluation complexity , 2012 .

[50]  Ya-xiang Yuan,et al.  On the convergence and worst-case complexity of trust-region and regularization methods for unconstrained optimization , 2015, Math. Program..

[51]  Jean-Pierre Dussault.  ARCq: a new adaptive regularization by cubics , 2018, Optim. Methods Softw..