Solving the Trust-Region Subproblem By a Generalized Eigenvalue Problem

The state-of-the-art algorithms for solving the trust-region subproblem (TRS) are based on an iterative process, involving solutions of many linear systems, eigenvalue problems, subspace optimization, or line search steps. A relatively underappreciated fact, due to Gander, Golub, and von Matt [Linear Algebra Appl., 114 (1989), pp. 815--839], is that TRSs can be solved by one generalized eigenvalue problem, with no outer iterations. In this paper we rediscover this fact and discover its great practicality, which exhibits good performance both in accuracy and efficiency. Moreover, we generalize the approach in various directions, namely by allowing for an ellipsoidal constraint, dealing with the so-called hard case, and obtaining approximate solutions efficiently when high accuracy is unnecessary. We demonstrate that the resulting algorithm is a general-purpose TRS solver, effective both for dense and large-sparse problems, including the so-called hard case. Our algorithm is easy to implement: its essence i...

[1]  Stephen J. Wright,et al.  Numerical Optimization , 2018, Fundamental Statistical Inference.

[2]  G. Forsythe,et al.  On the Stationary Values of a Second-Degree Polynomial on the Unit Sphere , 1965 .

[3]  Danny C. Sorensen,et al.  Algorithm 873: LSTRS: MATLAB software for large-scale trust-region subproblems and regularization , 2008, TOMS.

[4]  Satoru IWATA,et al.  A Polynomial-Time Algorithm for Nonconvex Quadratic Optimization with Two Quadratic Constraints , 2015 .

[5]  Stephen P. Boyd,et al.  Obstacle Collision Detection Using Best Ellipsoid Fit , 1997, J. Intell. Robotic Syst..

[6]  W. Gander,et al.  A constrained eigenvalue problem , 1988 .

[7]  Nicholas I. M. Gould,et al.  GALAHAD, a library of thread-safe Fortran 90 packages for large-scale nonlinear optimization , 2003, TOMS.

[8]  José Mario Martínez,et al.  Local Minimizers of Quadratic Functions on Euclidean Balls and Spheres , 1994, SIAM J. Optim..

[9]  Henry Wolkowicz,et al.  The generalized trust region subproblem , 2014, Comput. Optim. Appl..

[10]  Franz Rendl,et al.  A semidefinite framework for trust region subproblems with applications to large scale minimization , 1997, Math. Program..

[11]  G. Stewart,et al.  Matrix Perturbation Theory , 1990 .

[12]  David M. author-Gay.  Computing Optimal Locally Constrained Steps , 1981 .

[13]  Chao Yang,et al.  ARPACK users' guide - solution of large-scale eigenvalue problems with implicitly restarted Arnoldi methods , 1998, Software, environments, tools.

[14]  William W. Hager,et al.  Minimizing a Quadratic Over a Sphere , 2001, SIAM J. Optim..

[15]  T. Steihaug.  The Conjugate Gradient Method and Trust Regions in Large Scale Optimization , 1983 .

[16]  Galahad , 1906 .

[17]  Le Thi Hoai An,et al.  A D.C. Optimization Algorithm for Solving the Trust-Region Subproblem , 1998, SIAM J. Optim..

[18]  Henry Wolkowicz,et al.  The trust region subproblem and semidefinite programming , 2004, Optim. Methods Softw..

[19]  J. J. Moré.  Generalizations of the trust region problem , 1993 .

[20]  Nicholas I. M. Gould,et al.  Trust Region Methods , 2000, MOS-SIAM Series on Optimization.

[21]  Laura Palagi,et al.  On Some Properties of Quadratic Programs with a Convex Quadratic Constraint , 1998, SIAM J. Optim..

[22]  W. Gander,et al.  A D.C. OPTIMIZATION ALGORITHM FOR SOLVING THE TRUST-REGION SUBPROBLEM∗ , 1998 .

[23]  Samuel Burer,et al.  Second-Order-Cone Constraints for Extended Trust-Region Subproblems , 2013, SIAM J. Optim..

[24]  Greg Huber,et al.  Gamma Function Derivation of n-Sphere Volumes , 1982 .

[25]  Danny C. Sorensen,et al.  Minimization of a Large-Scale Quadratic FunctionSubject to a Spherical Constraint , 1997, SIAM J. Optim..

[26]  Gene H. Golub,et al.  Matrix computations , 1983 .

[27]  Elad Hazan,et al.  A linear-time algorithm for trust region problems , 2014, Math. Program..

[28]  Jorge J. Moré,et al.  Recent Developments in Algorithms and Software for Trust Region Methods , 1982, ISMP.

[29]  Philip E. Gill,et al.  Iterative Methods for Finding a Trust-region Step , 2009, SIAM J. Optim..

[30]  Philip E. Gill,et al.  A Subspace Minimization Method for the Trust-Region Step , 2009, SIAM J. Optim..

[31]  Akiko Takeda,et al.  Solving Generalized CDT Problems via Two-Parameter Eigenvalues , 2016, SIAM J. Optim..

[32]  Jack Dongarra,et al.  Templates for the Solution of Algebraic Eigenvalue Problems , 2000, Software, environments, tools.

[33]  Akiko Takeda,et al.  Global Optimization Methods for Extended Fisher Discriminant Analysis , 2014, AISTATS.

[34]  M. Saunders,et al.  Solution of Sparse Indefinite Systems of Linear Equations , 1975 .

[35]  Henk A. van der Vorst,et al.  Approximate solutions and eigenvalue bounds from Krylov subspaces , 1995, Numer. Linear Algebra Appl..

[36]  Jorge J. Moré,et al.  Computing a Trust Region Step , 1983 .

[37]  Nicholas I. M. Gould,et al.  Solving the Trust-Region Subproblem using the Lanczos Method , 1999, SIAM J. Optim..

[38]  Nicholas J. Higham,et al.  Functions of matrices - theory and computation , 2008 .

[39]  Nicholas I. M. Gould,et al.  On solving trust-region and other regularised subproblems in optimization , 2010, Math. Program. Comput..

[40]  Danny C. Sorensen,et al.  A New Matrix-Free Algorithm for the Large-Scale Trust-Region Subproblem , 2000, SIAM J. Optim..

[41]  Panayiotis E. Pintelas,et al.  A practical method for solving large-scale TRS , 2011, Optim. Lett..

[42]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[43]  Philippe L. Toint,et al.  Towards an efficient sparsity exploiting newton method for minimization , 1981 .