LSMR: An Iterative Algorithm for Sparse Least-Squares Problems

An iterative method LSMR is presented for solving linear systems $Ax=b$ and least-squares problems $\min \|Ax-b\|_2$, with $A$ being sparse or a fast linear operator. LSMR is based on the Golub-Kahan bidiagonalization process. It is analytically equivalent to the MINRES method applied to the normal equation $A^T\! Ax = A^T\! b$, so that the quantities $\|A^T\! r_k\|$ are monotonically decreasing (where $r_k = b - Ax_k$ is the residual for the current iterate $x_k$). We observe in practice that $\|r_k\|$ also decreases monotonically, so that compared to LSQR (for which only $\|r_k\|$ is monotonic) it is safer to terminate LSMR early. We also report some experiments with reorthogonalization.

[1]  D. Luenberger.  The Conjugate Residual Method for Constrained Minimization Problems , 1970 .

[2]  G. Stewart.  An inverse perturbation theorem for the linear least squares problem , 1975, SGNM.

[3]  M. Saunders,et al.  Solution of Sparse Indefinite Systems of Linear Equations , 1975 .

[4]  G. W. Stewart,et al.  Research, Development, and LINPACK , 1977 .

[5]  Michael A. Saunders,et al.  LSQR: An Algorithm for Sparse Linear Equations and Sparse Least Squares , 1982, TOMS.

[6]  Michael A. Saunders,et al.  Algorithm 583: LSQR: Sparse Linear Equations and Least Squares Problems , 1982, TOMS.

[7]  Y. Saad,et al.  GMRES: a generalized minimal residual algorithm for solving nonsymmetric linear systems , 1986 .

[8]  Nicholas J. Higham,et al.  INVERSE PROBLEMS NEWSLETTER , 1991 .

[9]  Ji-Guang Sun,et al.  Optimal backward perturbation bounds for the linear least squares problem , 1995, Numer. Linear Algebra Appl..

[10]  G. W. Stewart,et al.  The QLP Approximation to the Singular Value Decomposition , 1999, SIAM J. Sci. Comput..

[11]  Steven J. Benbow,et al.  Solving Generalized Least-Squares Problems with LSQR , 1999, SIAM J. Matrix Anal. Appl..

[12]  Z. Drmač,et al.  A new stable bidiagonal reduction algorithm , 2005 .

[13]  Miroslav Rozlozník,et al.  Modified Gram-Schmidt (MGS), Least Squares, and Backward Stability of MGS-GMRES , 2006, SIAM J. Matrix Anal. Appl..

[14]  Gene H. Golub,et al.  Calculating the singular values and pseudo-inverse of a matrix , 2007, Milestones in Matrix Computation.

[15]  J. Grcar,et al.  Estimates of Optimal Backward Perturbations for Linear Least Squares Problems , 2007 .

[16]  M. Arioli,et al.  Least-squares problems, normal equations, and stopping criteria for the conjugate gradient method , 2008 .

[17]  Xiao-Wen Chang,et al.  Stopping Criteria for the Iterative Solution of Linear Least Squares Problems , 2009, SIAM J. Matrix Anal. Appl..

[18]  Åke Björck,et al.  Least Squares Problems , 2009, Encyclopedia of Optimization.

[19]  Pavel Jiránek,et al.  Estimating the Backward Error in LSQR , 2010, SIAM J. Matrix Anal. Appl..

[20]  D. Titley-Péloquin.  Backward perturbation analysis of least squares problems , 2010 .

[21]  Jun-Feng Yin,et al.  GMRES Methods for Least Squares Problems , 2010, SIAM J. Matrix Anal. Appl..

[22]  Timothy A. Davis,et al.  The university of Florida sparse matrix collection , 2011, TOMS.

[23]  Michael A. Saunders,et al.  MINRES-QLP: A Krylov Subspace Method for Indefinite or Singular Symmetric Systems , 2010, SIAM J. Sci. Comput..

[24]  Serge Gratton,et al.  On the Accuracy of the Karlson-Waldén Estimate of the Backward Error for Linear Least Squares Problems , 2012, SIAM J. Matrix Anal. Appl..