Trust-region and other regularisations of linear least-squares problems

AbstractWe consider methods for regularising the least-squares solution of the linear system Ax=b. In particular, we propose iterative methods for solving large problems in which a trust-region bound ‖x‖≤Δ is imposed on the size of the solution, and in which the least value of linear combinations of ‖Ax−b‖2q and a regularisation term ‖x‖2p for various p and q=1,2 is sought. In each case, one or more “secular” equations are derived, and fast Newton-like solution procedures are suggested. The resulting algorithms are available as part of the $\mathsf{G}$ ALAHAD optimization library.

[1]  Kenneth Levenberg.  A METHOD FOR THE SOLUTION OF CERTAIN NON – LINEAR PROBLEMS IN LEAST SQUARES , 1944 .

[2]  D. D. Morrison.  Methods for nonlinear least squares problems and convergence proofs , 1960 .

[3]  D. Marquardt.  An Algorithm for Least-Squares Estimation of Nonlinear Parameters , 1963 .

[4]  James M. Ortega,et al.  Iterative solution of nonlinear equations in several variables , 2014, Computer science and applied mathematics.

[5]  C. Reinsch.  Smoothing by spline functions. II , 1971 .

[6]  M. D. Hebden,et al.  An algorithm for minimization using exact second derivatives , 1973 .

[7]  L. Eldén.  Algorithms for the regularization of ill-conditioned least squares problems , 1977 .

[8]  Jorge J. Moré,et al.  The Levenberg-Marquardt algo-rithm: Implementation and theory , 1977 .

[9]  J. Bunch,et al.  Rank-one modification of the symmetric eigenproblem , 1978 .

[10]  W. Gander.  Least squares with a quadratic constraint , 1980 .

[11]  Philippe L. Toint,et al.  Towards an efficient sparsity exploiting newton method for minimization , 1981 .

[12]  Michael A. Saunders,et al.  LSQR: An Algorithm for Sparse Linear Equations and Sparse Least Squares , 1982, TOMS.

[13]  Michael A. Saunders,et al.  Algorithm 583: LSQR: Sparse Linear Equations and Least Squares Problems , 1982, TOMS.

[14]  T. Steihaug.  The Conjugate Gradient Method and Trust Regions in Large Scale Optimization , 1983 .

[15]  I. Duff.  Sparse Matrices and Their Uses. , 1983 .

[16]  John E. Dennis,et al.  Numerical methods for unconstrained optimization and nonlinear equations , 1983, Prentice Hall series in computational mathematics.

[17]  P. Hansen.  Rank-Deficient and Discrete Ill-Posed Problems: Numerical Aspects of Linear Inversion , 1987 .

[18]  Josef Stoer,et al.  Numerische Mathematik 1 , 1989 .

[19]  E. Omojokun.  Trust region algorithms for optimization with nonlinear equality and inequality constraints , 1990 .

[20]  G. Golub,et al.  Quadratically constrained least squares and quadratic problems , 1991 .

[21]  Donald W. Cooley,et al.  Solving quadratically constrained least squares using black box solvers , 1992 .

[22]  Ladislav Luksan,et al.  Inexact trust region method for large sparse nonlinear least squares , 1993, Kybernetika.

[23]  A. Melman.  Numerical solution of a secular equation , 1995 .

[24]  J. Navarro-Pedreño.  Numerical Methods for Least Squares Problems , 1996 .

[25]  Jorge Nocedal,et al.  On the Implementation of an Algorithm for Large-Scale Equality Constrained Optimization , 1998, SIAM J. Optim..

[26]  Arnold Neumaier,et al.  Solving Ill-Conditioned and Singular Linear Systems: A Tutorial on Regularization , 1998, SIAM Rev..

[27]  Stephen J. Wright,et al.  Numerical Optimization , 2018, Fundamental Statistical Inference.

[28]  D. Sorensen,et al.  A large-scale trust-region approach to the regularization of discrete ill-posed problems , 1999 .

[29]  Nicholas I. M. Gould,et al.  Solving the Trust-Region Subproblem using the Lanczos Method , 1999, SIAM J. Optim..

[30]  Ya-Xiang Yuan,et al.  On the truncated conjugate gradient method , 2000, Math. Program..

[31]  Stephen J. Wright,et al.  Numerical Optimization (Springer Series in Operations Research and Financial Engineering) , 2000 .

[32]  William W. Hager,et al.  Minimizing a Quadratic Over a Sphere , 2001, SIAM J. Optim..

[33]  Danny C. Sorensen,et al.  A Trust-Region Approach to the Regularization of Large-Scale Discrete Forms of Ill-Posed Problems , 2001, SIAM J. Sci. Comput..

[34]  M. Heinkenschloss,et al.  Large-Scale PDE-Constrained Optimization , 2003 .

[35]  Y. Nesterov.  Modified Gauss-Newton Scheme with Worst-Case Guarantees for its Global Performance , 2003 .

[36]  Nicholas I. M. Gould,et al.  GALAHAD, a library of thread-safe Fortran 90 packages for large-scale nonlinear optimization , 2003, TOMS.

[37]  Nicholas I. M. Gould,et al.  CUTEr and SifDec: A constrained and unconstrained testing environment, revisited , 2003, TOMS.

[38]  Lorenz T. Biegler,et al.  A Trust Region SQP Algorithm for Equality Constrained Parameter Estimation with Simple Parameter Bounds , 2004, Comput. Optim. Appl..

[39]  William W. Hager,et al.  Global convergence of SSM for minimizing a quadratic over a sphere , 2004, Math. Comput..

[40]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[41]  Yurii Nesterov,et al.  Cubic regularization of Newton method and its global performance , 2006, Math. Program..

[42]  Yurii Nesterov,et al.  Modified Gauss–Newton scheme with worst case guarantees for global performance , 2007, Optim. Methods Softw..

[43]  Gene H. Golub,et al.  Calculating the singular values and pseudo-inverse of a matrix , 2007, Milestones in Matrix Computation.

[44]  P. Toint,et al.  Adaptive cubic overestimation methods for unconstrained optimization , 2007 .

[45]  Philip E. Gill,et al.  Iterative Methods for Finding a Trust-region Step , 2009, SIAM J. Optim..

[46]  Nicholas I. M. Gould,et al.  Nonlinear programming without a penalty function or a filter , 2010, Math. Program..