The Lanczos and conjugate gradient algorithms in finite precision arithmetic

The Lanczos and conjugate gradient algorithms were introduced more than five decades ago as tools for numerical computation of dominant eigenvalues of symmetric matrices and for solving linear algebraic systems with symmetric positive definite matrices, respectively. Because of their fundamental relationship with the theory of orthogonal polynomials and Gauss quadrature of the Riemann-Stieltjes integral, the Lanczos and conjugate gradient algorithms represent very interesting general mathematical objects, with highly nonlinear properties which can be conveniently translated from algebraic language into the language of mathematical analysis, and vice versa. The algorithms are also very interesting numerically, since their numerical behaviour can be explained by an elegant mathematical theory, and the interplay between analysis and algebra is useful there too. Motivated by this view, the present contribution wishes to pay a tribute to those who have made an understanding of the Lanczos and conjugate gradient algorithms possible through their pioneering work, and to review recent solutions of several open problems that have also contributed to knowledge of the subject.

[1]  É. Christoffel,et al.  Sur une classe particulière de fonctions entières et de fractions continues , 2022 .

[2]  G. Darboux,et al.  Mémoire sur l'approximation des fonctions de très-grands nombres, et sur une classe étendue de développements en série. , 1878 .

[3]  C. Lanczos.  An iteration method for the solution of the eigenvalue problem of linear differential and integral operators , 1950 .

[4]  W. Arnoldi.  The principle of minimized iterations in the solution of the matrix eigenvalue problem , 1951 .

[5]  M. Hestenes,et al.  Methods of conjugate gradients for solving linear systems , 1952 .

[6]  C. Lanczos.  Solution of Systems of Linear Equations by Minimized Iterations1 , 1952 .

[7]  V. Fridman,et al.  The method of minimum iterations with minimum errors for a system of linear algebraic equations with a symmetrical matrix , 1963 .

[8]  G. Meinardus,et al.  Über eine Verallgemeinerung einer Ungleichung von L. V. Kantorowitsch , 1963 .

[9]  Alston S. Householder,et al.  The Theory of Matrices in Numerical Analysis , 1964 .

[10]  THE THEORY OF MATRICES IN NUMERICAL ANALYSIS , 1965 .

[11]  J. H. Wilkinson.  The algebraic eigenvalue problem , 1966 .

[12]  S. Kaniel.  Estimates for Some Computational Techniques - in Linear Algebra , 1966 .

[13]  Philip Rabinowitz,et al.  Methods of Numerical Integration , 1985 .

[14]  J. Daniel.  The Conjugate Gradient Method for Linear and Nonlinear Operator Equations , 1967 .

[15]  W. Gautschi.  Construction of Gauss-Christoffel quadrature formulas , 1968 .

[16]  R. C. Thompson,et al.  Principal submatrices II: the upper and lower quadratic inequalities☆ , 1968 .

[17]  D. Luenberger.  Hyperbolic Pairs in the Method of Conjugate Gradients , 1969 .

[18]  Christopher C. Paige,et al.  Practical use of the symmetric Lanczos process with re-orthogonalization , 1970 .

[19]  D. Luenberger.  The Conjugate Residual Method for Constrained Minimization Problems , 1970 .

[20]  Christopher C. Paige,et al.  The computation of eigenvalues and eigenvectors of very large sparse matrices , 1971 .

[21]  C. Paige.  Computational variants of the Lanczos method for the eigenproblem , 1972 .

[22]  G. Golub,et al.  Bounds for the error of linear systems of equations using the theory of moments , 1972 .

[23]  Christopher C. Paige,et al.  Eigenvalues of perturbed Hermitian matrices , 1974 .

[24]  M. Saunders,et al.  Solution of Sparse Indefinite Systems of Linear Equations , 1975 .

[25]  C. Paige.  Error Analysis of the Lanczos Algorithm for Tridiagonalizing a Symmetric Matrix , 1976 .

[26]  R. Fletcher.  Conjugate gradient methods for indefinite systems , 1976 .

[27]  J. G. Lewis.  Algorithms for sparse matrix eigenvalue problems , 1977 .

[28]  H Woz╠üniakowski.  Round-off error analysis of iterations for large linear systems , 1977 .

[29]  A. Greenbaum.  Comparison of splittings used with the conjugate gradient algorithm , 1979 .

[30]  B. Parlett,et al.  The Lanczos algorithm with selective orthogonalization , 1979 .

[31]  D. S. Scott,et al.  How to make the Lanczos algorithm converge slowly , 1979 .

[32]  G. Golub,et al.  Bounds for the error in linear systems , 1979 .

[33]  C. Paige.  Accuracy and effectiveness of the Lanczos algorithm for the symmetric eigenproblem , 1980 .

[34]  Henryk Woźniakowski.  Roundoff-error analysis of a new class of conjugate-gradient algorithms , 1980 .

[35]  B. Parlett.  The Symmetric Eigenvalue Problem , 1981 .

[36]  Y. Saad.  On the Rates of Convergence of the Lanczos and the Block-Lanczos Methods , 1980 .

[37]  J. Bollen.  Round-off error analysis of descent methods for solving linear equations , 1980 .

[38]  W. Gautschi.  A Survey of Gauss-Christoffel Quadrature Formulae , 1981 .

[39]  J. Grcar.  Analyses of the lanczos algorithm and of the approximation problem in richardson's method , 1981 .

[40]  Anne Greenbaum.  Convergence properties of the conjugate gradient algorithm in exact and finite precision arithmetic , 1981 .

[41]  W. Gautschi.  On Generating Orthogonal Polynomials , 1982 .

[42]  H. Simon.  The lanczos algorithm for solving symmetric linear systems , 1982 .

[43]  Josef Stoer,et al.  Solution of Large Linear Systems of Equations by Conjugate Gradient Type Methods , 1982, ISMP.

[44]  Gene H. Golub,et al.  On the calculation of Jacobi Matrices , 1983 .

[45]  W. Gragg,et al.  The numerically stable reconstruction of Jacobi matrices from spectral data , 1984 .

[46]  H. Simon.  Analysis of the symmetric Lanczos algorithm with reorthogonalization methods , 1984 .

[47]  F. R. Gantmakher.  The Theory of Matrices , 1984 .

[48]  H. Simon.  The Lanczos algorithm with partial reorthogonalization , 1984 .

[49]  Jo A. Bollen.  Numerical stability of descent methods for solving linear equations , 1984 .

[50]  J. Cullum,et al.  Lanczos algorithms for large symmetric eigenvalue computations , 1985 .

[51]  W. Gautschi.  Orthogonal polynomials-Constructive theory and applications * , 1985 .

[52]  O. Axelsson,et al.  On the rate of convergence of the preconditioned conjugate gradient method , 1986 .

[53]  H. V. D. Vorst,et al.  The rate of convergence of Conjugate Gradients , 1986 .

[54]  H. V. D. Vorst,et al.  The convergence behavior of ritz values in the presence of close eigenvalues , 1987 .

[55]  Gene H. Golub,et al.  Some History of the Conjugate Gradient and Lanczos Algorithms: 1948-1976 , 1989, SIAM Rev..

[56]  A. Greenbaum.  Behavior of slightly perturbed Lanczos and conjugate-gradient recurrences , 1989 .

[57]  Beresford N. Parlett.  The contribution of J. H. Wilkinson to numerical analysis , 1990 .

[58]  Nicholas J. Higham,et al.  INVERSE PROBLEMS NEWSLETTER , 1991 .

[59]  G. Golub,et al.  Quadratically constrained least squares and quadratic problems , 1991 .

[60]  Z. Strakos,et al.  On the real convergence rate of the conjugate gradient method , 1991 .

[61]  Anne Greenbaum,et al.  Predicting the Behavior of Finite Precision Lanczos and Conjugate Gradient Computations , 2015, SIAM J. Matrix Anal. Appl..

[62]  Zdeněk Strakoš,et al.  Open questions in the convergence analysis of the lanczos process for the real symmetric eigenvalue problem , 1992 .

[63]  L. Knizhnerman,et al.  Error bounds in the simple Lanczos procedure for computing functions of symmetric matrices and eigenvalues , 1992 .

[64]  P. Deuflhard.  Cascadic conjugate gradient methods for elliptic partial differential equations , 1993 .

[65]  W. Hackbusch.  Iterative Solution of Large Sparse Systems of Equations , 1993 .

[66]  L. Knizhnerman.  The quality of approximations to an isolated eigenvalue and the distribution of “Ritz numbers” in the sample Lanczos procedure , 1995 .

[67]  Qiang Ye.  On close eigenvalues of tridiagonal matrices , 1995 .

[68]  B. Parlett.  Do We Fully Understand the Symmetric Lanczos Algorithm Yet , 1995 .

[69]  Henk A. van der Vorst,et al.  Approximate solutions and eigenvalue bounds from Krylov subspaces , 1995, Numer. Linear Algebra Appl..

[70]  L. Knizhnerman.  The simple Lanczos procedure: estimates of the error of the Gauss quadrature formula and their applications , 1996 .

[71]  Beresford N. Parlett,et al.  Invariant subspaces for tightly clustered eigenvalues of tridiagonals , 1996 .

[72]  B. Fischer.  Polynomial Based Iteration Methods for Symmetric Linear Systems , 1996 .

[73]  Anne Greenbaum,et al.  Iterative methods for solving linear systems , 1997, Frontiers in applied mathematics.

[74]  G. Golub,et al.  Matrices, moments and quadrature II; How to compute the norm of the error in iterative methods , 1997 .

[75]  A. Greenbaum.  Estimating the Attainable Accuracy of Recursively Computed Residual Methods , 1997, SIAM J. Matrix Anal. Appl..

[76]  G. W. Stewart,et al.  Matrix algorithms , 1998 .

[77]  Å. Björck,et al.  Stability of Conjugate Gradient and Lanczos Methods for Linear Least Squares Problems , 1998, SIAM J. Matrix Anal. Appl..

[78]  Anne Greenbaum,et al.  Using Nonorthogonal Lanczos Vectors in the Computation of Matrix Functions , 1998, SIAM J. Sci. Comput..

[79]  Accuracy of two three-term and three two-term recurrences for Krylov space solvers , 1999 .

[80]  Gene H. Golub,et al.  On Some Eigenvector-Eigenvalue Relations , 1999, SIAM J. Matrix Anal. Appl..

[81]  Dirk P. Laurie.  Accurate recovery of recursion coe cients from Gaussian quadrature formulas , 1999 .

[82]  G. Meurant.  Computer Solution of Large Linear Systems , 1999 .

[83]  B. Parlett,et al.  Relatively robust representations of symmetric tridiagonals , 2000 .

[84]  Zdenek Strakos,et al.  Accuracy of Two Three-term and Three Two-term Recurrences for Krylov Space Solvers , 2000, SIAM J. Matrix Anal. Appl..

[85]  Dirk Laurie,et al.  Computation of Gauss-type quadrature formulas , 2001 .

[86]  Owe Axelsson,et al.  Error norm estimation and stopping criteria in preconditioned conjugate gradient iterations , 2001, Numer. Linear Algebra Appl..

[87]  M. Arioli,et al.  Stopping criteria for iterative methods:¶applications to PDE's , 2001 .

[88]  Michael L. Overton,et al.  Numerical Computing with IEEE Floating Point Arithmetic , 2001 .

[89]  G. Sleijpen,et al.  Inexact Krylov Subspace Methods for Linear Systems , 2004, SIAM J. Matrix Anal. Appl..

[90]  J. Stoer,et al.  Introduction to Numerical Analysis , 2002 .

[91]  Walter Gautschi,et al.  THE INTERPLAY BETWEEN CLASSICAL ANALYSIS AND (NUMERICAL) LINEAR ALGEBRA — A TRIBUTE TO GENE H. GOLUB , 2002 .

[92]  Arieh Iserles,et al.  On the Foundations of Computational Mathematics , 2001 .

[93]  Z. Strakos,et al.  On error estimation in the conjugate gradient method and why it works in finite precision computations. , 2002 .

[94]  Arno B. J. Kuijlaars,et al.  Superlinear CG convergence for special right-hand sides , 2002 .

[95]  W. Gautschi,et al.  THE INTERPLAY BETWEEN CLASSICAL ANALYSIS AND ( NUMERICAL ) LINEAR ALGEBRA — A TRIBUTE TO GENE , 2002 .

[96]  Jens-Peter M. Zemke,et al.  Krylov Subspace Methods in Finite Precision : A Unified Approach , 2003 .

[97]  Inderjit S. Dhillon,et al.  Orthogonal Eigenvectors and Relative Gaps , 2003, SIAM J. Matrix Anal. Appl..

[98]  Gérard Meurant.  The computation of bounds for the norm of the error in the conjugate gradient algorithm , 2004, Numerical Algorithms.

[99]  Gérard Meurant,et al.  Numerical experiments in computing bounds for the norm of the error in the preconditioned conjugate gradient algorithm , 1999, Numerical Algorithms.

[100]  David M. Young,et al.  Applied Iterative Methods , 2004 .

[101]  M. Arioli,et al.  A stopping criterion for the conjugate gradient algorithm in a finite element method framework , 2000, Numerische Mathematik.

[102]  W. Gautschi.  Orthogonal Polynomials: Computation and Approximation , 2004 .

[103]  B. Parlett,et al.  Multiple representations to compute orthogonal eigenvectors of symmetric tridiagonal matrices , 2004 .

[104]  Serena Morigi,et al.  Computable error bounds and estimates for the conjugate gradient method , 2000, Numerical Algorithms.

[105]  Gerard L. G. Sleijpen,et al.  BiCGstab(l) and other hybrid Bi-CG methods , 1994, Numerical Algorithms.

[106]  JÖRG LIESEN,et al.  ON THE WORST-CASE CONVERGENCE OF MR AND CG FOR SYMMETRIC POSITIVE DEFINITE TRIDIAGONAL TOEPLITZ MATRICES , 2005 .

[107]  Valérie Frayssé,et al.  Inexact Matrix-Vector Products in Krylov Methods for Solving Linear Systems: A Relaxation Strategy , 2005, SIAM J. Matrix Anal. Appl..

[108]  W. Wülling,et al.  The Stabilization of Weights in the Lanczos and Conjugate Gradient Method , 2005 .

[109]  Gérard Meurant.  Estimates of the l2 norm of the error in the conjugate gradient algorithm , 2005, Numerical Algorithms.

[110]  Numerische,et al.  On the convergence rate of the conjugate gradients in presence of rounding errors * , 2005 .

[111]  Liliana Borcea,et al.  On the sensitivity of Lanczos recursions to the spectrum , 2005 .

[112]  Gene H. Golub,et al.  Estimates in quadratic formulas , 1994, Numerical Algorithms.

[113]  Z. Strakos,et al.  Error Estimation in Preconditioned Conjugate Gradients , 2005 .

[114]  Wolfgang Wülling.  On Stabilization and Convergence of Clustered Ritz Values in the Lanczos Method , 2005, SIAM J. Matrix Anal. Appl..

[115]  Z. Strakos,et al.  On numerical stability in large scale linear algebraic computations , 2005 .

[116]  Arno B. J. Kuijlaars,et al.  Convergence Analysis of Krylov Subspace Iterations with Methods from Potential Theory , 2006, SIAM Rev..

[117]  G. Meurant.  The Lanczos and Conjugate Gradient Algorithms: From Theory to Finite Precision Computations , 2006 .

[118]  Gene H. Golub,et al.  Calculation of Gauss quadrature rules , 1967, Milestones in Matrix Computation.

[119]  Petr Tichý,et al.  On sensitivity of Gauss–Christoffel quadrature , 2007, Numerische Mathematik.

[120]  Gene H. Golub,et al.  Matrices, moments, and quadrature , 2007, Milestones in Matrix Computation.

[121]  Ren-Cang Li,et al.  On Meinardus' examples for the conjugate gradient method , 2008, Math. Comput..

[122]  G. Meurant.  The Lanczos and conjugate gradient algorithms , 2008 .

[123]  Z. Strakos,et al.  ON SENSITIVITY OF GAUSS-CHRISTOFFEL QUADRATURE ESTIMATES∗ , 2022 .