Least Squares Residuals and Minimal Residual Methods

We study Krylov subspace methods for solving unsymmetric linear algebraic systems that minimize the norm of the residual at each step (minimal residual (MR) methods). MR methods are often formulated in terms of a sequence of least squares (LS) problems of increasing dimension. We present several basic identities and bounds for the LS residual. These results are interesting in the general context of solving LS problems. When applied to MR methods, they show that the size of the MR residual is strongly related to the conditioning of different bases of the same Krylov subspace. Using different bases is useful in theory because relating convergence to the characteristics of different bases offers new insight into the behavior of MR methods. Different bases also lead to different implementations which are mathematically equivalent but can differ numerically. Our theoretical results are used for a finite precision analysis of implementations of the GMRES method [Y. Saad and M. H. Schultz, SIAM J. Sci. Statist. Comput., 7 (1986), pp. 856--869]. We explain that the choice of the basis is fundamental for the numerical stability of the implementation. As demonstrated in the case of Simpler GMRES [H. F. Walker and L. Zhou, Numer. Linear Algebra Appl., 1 (1994), pp. 571--581], the best orthogonalization technique used for computing the basis does not compensate for the loss of accuracy due to an inappropriate choice of the basis. In particular, we prove that Simpler GMRES is inherently less numerically stable than the Classical GMRES implementation due to Saad and Schultz [SIAM J. Sci. Statist. Comput., 7 (1986), pp. 856--869].

[1]  L. Reichel,et al.  A Newton basis GMRES implementation , 1994 .

[2]  H. Walker.  Implementation of the GMRES method using householder transformations , 1988 .

[3]  Ilse C. F. Ipsen.  Expressions and Bounds for the GMRES Residual , 2000, Bit Numerical Mathematics.

[4]  Jocelyne Erhel,et al.  A parallel GMRES version for general sparse matrices. , 1995 .

[5]  H. V. D. Vorst,et al.  Reducing the effect of global communication in GMRES( m ) and CG on parallel distributed memory computers , 1995 .

[6]  Gene H. Golub,et al.  Matrix computations (3rd ed.) , 1996 .

[7]  Y. Saad,et al.  GMRES: a generalized minimal residual algorithm for solving nonsymmetric linear systems , 1986 .

[8]  J. Navarro-Pedreño.  Numerical Methods for Least Squares Problems , 1996 .

[9]  Homer F. Walker,et al.  Implementations of the GMRES method , 1989 .

[10]  Zdenek Strakos,et al.  Residual and Backward Error Bounds in Minimum Residual Krylov Subspace Methods , 2001, SIAM J. Sci. Comput..

[11]  H. Elman.  Iterative methods for large, sparse, nonsymmetric systems of linear equations , 1982 .

[12]  S. Eisenstat,et al.  Variational Iterative Methods for Nonsymmetric Systems of Linear Equations , 1983 .

[13]  Kang C. Jea,et al.  Generalized conjugate-gradient acceleration of nonsymmetrizable iterative methods , 1980 .

[14]  M. Rozložník,et al.  Numerical stability of GMRES , 1995 .

[15]  Christopher C. Paige,et al.  Scaled total least squares fundamentals , 2002, Numerische Mathematik.

[16]  Roger B. Sidje,et al.  Alternatives for parallel Krylov subspace basis computation , 1997, Numer. Linear Algebra Appl..

[17]  Jörg Liesen,et al.  Construction and analysis of polynomial iterative methods for non-hermitian systems of linear equations , 1998 .

[18]  Ilse C. F. Ipsen.  A different approach to bounding the minimal residual norm in Krylov methods , 1998 .

[19]  Ricardo D. Fierro,et al.  The Total Least Squares Problem: Computational Aspects and Analysis (S. Van Huffel and J. Vandewalle) , 1993, SIAM Rev..

[20]  Mei Han An,et al.  accuracy and stability of numerical algorithms , 1991 .

[21]  G. Stewart.  Collinearity and Least Squares Regression , 1987 .

[22]  Sabine Van Huffel,et al.  Total least squares problem - computational aspects and analysis , 1991, Frontiers in applied mathematics.

[23]  M. Rozložník,et al.  Numerical behaviour of the modified gram-schmidt GMRES implementation , 1997 .

[24]  O. Axelsson.  A generalized conjugate gradient, least square method , 1987 .

[25]  C. D. Meyer,et al.  Generalized inverses of linear transformations , 1979 .

[26]  O. Axelsson.  Conjugate gradient type methods for unsymmetric and inconsistent systems of linear equations , 1980 .

[27]  H. V. D. Vorst,et al.  A comparison of some GMRES-like methods , 1992 .

[28]  I. M. Khabaza.  An Iterative Least-Square Method Suitable for Solving Large Sparse Matrices , 1963, Comput. J..

[29]  Homer F. Walker,et al.  A simpler GMRES , 1994, Numer. Linear Algebra Appl..

[30]  P. K. W. Vinsome,et al.  Orthomin, an Iterative Method for Solving Sparse Sets of Simultaneous Linear Equations , 1976 .

[31]  E. Sturler.  A PARALLEL VARIANT OF GMRES(m) , 1991 .