Error estimates for iterative algorithms for minimizing regularized quadratic subproblems

ABSTRACT We derive bounds for the objective errors and gradient residuals when finding approximations to the solution of common regularized quadratic optimization problems within evolving Krylov spaces. These provide upper bounds on the number of iterations required to achieve a given stated accuracy. We illustrate the quality of our bounds on given test examples.

[1]  Yair Carmon,et al.  Analysis of Krylov Subspace Solutions of Regularized Non-Convex Quadratic Problems , 2018, NeurIPS.

[2]  W. Marsden.  I and J , 2012 .

[3]  Henry Wolkowicz,et al.  The trust region subproblem and semidefinite programming , 2004, Optim. Methods Softw..

[4]  D K Smith,et al.  Numerical Optimization , 2001, J. Oper. Res. Soc..

[5]  Philip E. Gill,et al.  Iterative Methods for Finding a Trust-region Step , 2009, SIAM J. Optim..

[6]  Nicholas I. M. Gould,et al.  On solving trust-region and other regularised subproblems in optimization , 2010, Math. Program. Comput..

[7]  Danny C. Sorensen,et al.  A New Matrix-Free Algorithm for the Large-Scale Trust-Region Subproblem , 2000, SIAM J. Optim..

[8]  Philip E. Gill,et al.  A Subspace Minimization Method for the Trust-Region Step , 2009, SIAM J. Optim..

[9]  Are Magnus Bruaset,et al.  Krylov Subspace Methods , 2018, Krylov Subspace Methods with Application in Incompressible Fluid Flow Solvers.

[10]  Anne Greenbaum,et al.  Iterative methods for solving linear systems , 1997, Frontiers in applied mathematics.

[11]  Yong Xia,et al.  A linear-time algorithm for the trust region subproblem based on hidden convexity , 2017, Optim. Lett..

[12]  Owe Axelsson,et al.  Reaching the superlinear convergence phase of the CG method , 2014, J. Comput. Appl. Math..

[13]  Ren-Cang Li,et al.  On the Generalized Lanczos Trust-Region Method , 2017, SIAM J. Optim..

[14]  Nicholas I. M. Gould,et al.  CUTEst: a Constrained and Unconstrained Testing Environment with safe threads for mathematical optimization , 2013, Computational Optimization and Applications.

[15]  Philippe L. Toint,et al.  Towards an efficient sparsity exploiting newton method for minimization , 1981 .

[16]  David M. author-Gay.  Computing Optimal Locally Constrained Steps , 1981 .

[17]  William W. Hager,et al.  Minimizing a Quadratic Over a Sphere , 2001, SIAM J. Optim..

[18]  Yair Carmon,et al.  Gradient Descent Efficiently Finds the Cubic-Regularized Non-Convex Newton Step , 2016, ArXiv.

[19]  Nicholas I. M. Gould,et al.  Solving the Trust-Region Subproblem using the Lanczos Method , 1999, SIAM J. Optim..

[20]  José Mario Martínez,et al.  Local Minimizers of Quadratic Functions on Euclidean Balls and Spheres , 1994, SIAM J. Optim..

[21]  Franz Rendl,et al.  A semidefinite framework for trust region subproblems with applications to large scale minimization , 1997, Math. Program..

[22]  Jorge J. Moré,et al.  Computing a Trust Region Step , 1983 .

[23]  William F. Moss,et al.  Decay rates for inverses of band matrices , 1984 .

[24]  Chungen Shen,et al.  A Nested Lanczos Method for the Trust-Region Subproblem , 2018, SIAM J. Sci. Comput..

[25]  Barbara Kaltenbacher,et al.  Iterative Solution Methods , 2015, Handbook of Mathematical Methods in Imaging.

[26]  Danny C. Sorensen,et al.  Accelerating the LSTRS Algorithm , 2010, SIAM J. Sci. Comput..

[27]  Amir Beck,et al.  Globally Solving the Trust Region Subproblem Using Simple First-Order Methods , 2018, SIAM J. Optim..

[28]  L. Lukšan,et al.  On Lagrange multipliers of trust-region subproblems , 2008 .

[29]  Nam Ho-Nguyen,et al.  A Second-Order Cone Based Approach for Solving the Trust-Region Subproblem and Its Variants , 2016, SIAM J. Optim..

[30]  Nicholas I. M. Gould,et al.  Adaptive cubic regularisation methods for unconstrained optimization. Part I: motivation, convergence and numerical results , 2011, Math. Program..

[31]  Owe Axelsson,et al.  On the sublinear and superlinear rate of convergence of conjugate gradient methods , 2000, Numerical Algorithms.

[32]  Danny C. Sorensen,et al.  Algorithm 873: LSTRS: MATLAB software for large-scale trust-region subproblems and regularization , 2008, TOMS.

[33]  Nicholas I. M. Gould,et al.  GALAHAD, a library of thread-safe Fortran 90 packages for large-scale nonlinear optimization , 2003, TOMS.

[34]  T. Steihaug.  The Conjugate Gradient Method and Trust Regions in Large Scale Optimization , 1983 .

[35]  Yurii Nesterov,et al.  Cubic regularization of Newton method and its global performance , 2006, Math. Program..

[36]  Akiko Takeda,et al.  Solving the Trust-Region Subproblem By a Generalized Eigenvalue Problem , 2017, SIAM J. Optim..

[37]  Elad Hazan,et al.  A linear-time algorithm for trust region problems , 2014, Math. Program..

[38]  Stephen J. Wright,et al.  Complexity Analysis of Second-Order Line-Search Algorithms for Smooth Nonconvex Optimization , 2017, SIAM J. Optim..

[39]  Marco Sciandrone,et al.  On the use of iterative methods in cubic regularization for unconstrained optimization , 2015, Comput. Optim. Appl..