How Fast are Nonsymmetric Matrix Iterations?

Three leading iterative methods for the solution of nonsymmetric systems of linear equations are CGN (the conjugate gradient iteration applied to the normal equations), GMRES (residual minimization in a Krylov space), and CGS (a biorthogonalization algorithm adapted from the biconjugate gradient iteration). Do these methods differ fundamentally in capabilities? If so, which is best under which circumstances? The existing literature, in relying mainly on empirical studies, has failed to confront these questions systematically. In this paper it is shown that the convergence of CGN is governed by singular values and that of GMRES and CGS by eigenvalues or pseudo-eigenvalues. The three methods are found to be fundamentally different, and to substantiate this conclusion, examples of matrices are presented for which each iteration outperforms the others by a factor of size $O(\sqrt N )$ or $O(N)$ where N is the matrix dimension. Finally, it is shown that the performance of iterative methods for a particular mat...

[1]  M. Hestenes,et al.  Methods of conjugate gradients for solving linear systems , 1952 .

[2]  C. Lanczos.  Solution of Systems of Linear Equations by Minimized Iterations1 , 1952 .

[3]  R. Fletcher.  Conjugate gradient methods for indefinite systems , 1976 .

[4]  A. Greenbaum.  Comparison of splittings used with the conjugate gradient algorithm , 1979 .

[5]  C. Brezinski.  Padé-type approximation and general orthogonal polynomials , 1980 .

[6]  Y. Saad.  The Lanczos Biorthogonalization Algorithm and Other Oblique Projection Methods for Solving Large Unsymmetric Systems , 1982 .

[7]  Michael A. Saunders,et al.  LSQR: An Algorithm for Sparse Linear Equations and Sparse Least Squares , 1982, TOMS.

[8]  H. Elman.  Iterative methods for large, sparse, nonsymmetric systems of linear equations , 1982 .

[9]  S. Eisenstat,et al.  Variational Iterative Methods for Nonsymmetric Systems of Linear Equations , 1983 .

[10]  Y. Saad,et al.  Conjugate gradient-like algorithms for solving nonsymmetric linear systems , 1985 .

[11]  H. Elman,et al.  Polynomial iteration for nonsymmetric indefinite linear systems , 1986 .

[12]  H. V. D. Vorst,et al.  The rate of convergence of Conjugate Gradients , 1986 .

[13]  A. Edelman.  Eigenvalues and condition numbers of random matrices , 1988 .

[14]  H. Simon,et al.  Two Conjugate-Gradient-Type Methods for Unsymmetric Linear Equations , 1988 .

[15]  S. Succi,et al.  Iterative algorithms for the solution of nonsymmetric systems in the modelling of weak plasma turbulence , 1989 .

[16]  P. Sonneveld.  CGS, A Fast Lanczos-Type Solver for Nonsymmetric Linear systems , 1989 .

[17]  Y. Saad,et al.  Krylov Subspace Methods on Supercomputers , 1989 .

[18]  W. Joubert.  Generalized conjugate gradient and Lanczos methods for the solution of nonsymmetric systems of linear equations , 1990 .

[19]  T. Manteuffel,et al.  A taxonomy for conjugate gradient methods , 1990 .

[20]  L. Trefethen.  Approximation theory and numerical linear algebra , 1990 .

[21]  Peter N. Brown,et al.  A Theoretical Comparison of the Arnoldi and GMRES Algorithms , 1991, SIAM J. Sci. Comput..

[22]  Henk A. van der Vorst,et al.  Bi-CGSTAB: A Fast and Smoothly Converging Variant of Bi-CG for the Solution of Nonsymmetric Linear Systems , 1992, SIAM J. Sci. Comput..

[23]  Roland W. Freund,et al.  Conjugate Gradient-Type Methods for Linear Systems with Complex Symmetric Coefficient Matrices , 1992, SIAM J. Sci. Comput..

[24]  Lloyd N. Trefethen,et al.  A Hybrid GMRES Algorithm for Nonsymmetric Linear Systems , 1992, SIAM J. Matrix Anal. Appl..

[25]  G. Golub,et al.  Iterative solution of linear systems , 1991, Acta Numerica.

[26]  Roland W. Freund,et al.  An Implementation of the Look-Ahead Lanczos Algorithm for Non-Hermitian Matrices , 1993, SIAM J. Sci. Comput..

[27]  G. Golub,et al.  Gmres: a Generalized Minimum Residual Algorithm for Solving , 2022 .