Complexity of Projected Newton Methods for Bound-constrained Optimization

We analyze the iteration complexity of two methods based on the projected gradient and Newton methods for solving bound-constrained optimization problems. The first method is a scaled variant of Bertsekas’s twometric projection method [2], which can be shown to output an -approximate first-order point in O( −2) iterations. The second is a projected NewtonConjugate Gradient (CG) method, which locates an -approximate secondorder point with high probability in O( −3/2) iterations, at a cost of O( −7/4) gradient evaluations or Hessian-vector products (omitting logarithmic factors). Besides having good complexity properties, both methods are appealing from a practical point of view, as we show using some illustrative numerical results.

[1]  Xiaojun Chen,et al.  Complexity analysis of interior point algorithms for non-Lipschitz and nonconvex minimization , 2015, Math. Program..

[2]  Stephen J. Wright,et al.  A log-barrier Newton-CG method for bound constrained optimization with complexity guarantees , 2019, IMA Journal of Numerical Analysis.

[3]  Stephen J. Wright,et al.  A Newton-CG algorithm with complexity guarantees for smooth unconstrained optimization , 2018, Mathematical Programming.

[4]  Gerardo Toraldo,et al.  On the Solution of Large Quadratic Programming Problems with Bound Constraints , 1991, SIAM J. Optim..

[5]  Nicolas Gillis,et al.  The Why and How of Nonnegative Matrix Factorization , 2014, ArXiv.

[6]  Stephen J. Wright,et al.  Complexity Analysis of Second-Order Line-Search Algorithms for Smooth Nonconvex Optimization , 2017, SIAM J. Optim..

[7]  Changshui Zhang,et al.  Efficient Nonnegative Matrix Factorization via projected Newton method , 2012, Pattern Recognit..

[8]  Haesun Park,et al.  Toward Faster Nonnegative Matrix Factorization: A New Algorithm and Comparisons , 2008, 2008 Eighth IEEE International Conference on Data Mining.

[9]  D. Bertsekas.  Projected Newton methods for optimization problems with simple constraints , 1981, 1981 20th IEEE Conference on Decision and Control including the Symposium on Adaptive Processes.

[10]  Dimitri P. Bertsekas,et al.  Constrained Optimization and Lagrange Multiplier Methods , 1982 .

[11]  Yinyu Ye,et al.  Optimality condition and complexity analysis for linearly-constrained optimization without differentiability on the boundary , 2017, Math. Program..

[12]  Chih-Jen Lin,et al.  Projected Gradient Methods for Nonnegative Matrix Factorization , 2007, Neural Computation.

[13]  P. Toint,et al.  An adaptive cubic regularization algorithm for nonconvex optimization with convex constraints and its function-evaluation complexity , 2012 .