70 years of Krylov subspace methods: The journey continues

. Using computed examples for the Conjugate Gradient method and GMRES, we recall important building blocks in the understanding of Krylov subspace methods over the last 70 years. Each example consists of a description of the setup and the numerical observations, followed by an explanation of the observed phenomena, where we keep technical details as small as possible. Our goal is to show the mathematical beauty and hidden intricacies of the methods, and to point out some persistent misunderstandings as well as important open problems. We hope that this work initiates further investigations of Krylov subspace methods, which are efficient computational tools and exciting mathematical objects that are far from being fully understood.

[1]  B. F. Nielsen,et al.  Numerical approximation of the spectrum of self-adjoint operators in operator preconditioning , 2022, Numerical Algorithms.

[2]  Gérard Meurant,et al.  Accurate error estimation in CG , 2021, Numerical Algorithms.

[3]  Pseudospectra of matrices , 2020, Spectra and Pseudospectra.

[4]  L. Trefethen,et al.  Spectra and Pseudospectra , 2020 .

[5]  J. Tropp,et al.  Randomized numerical linear algebra: Foundations and algorithms , 2020, Acta Numerica.

[6]  Erin Carson,et al.  On the cost of iterative computations , 2020, Philosophical Transactions of the Royal Society A.

[7]  G. Meurant.  On prescribing the convergence behavior of the conjugate gradient algorithm , 2019, Numerical Algorithms.

[8]  Thomas Trogdon,et al.  Universality in numerical computation with random data: Case studies and analytical results , 2019, Journal of Mathematical Physics.

[9]  Z. Strakos,et al.  Algebraic description of the finite Stieltjes moment problem , 2019, Linear Algebra and its Applications.

[10]  Are Magnus Bruaset,et al.  Krylov Subspace Methods , 2018, Krylov Subspace Methods with Application in Incompressible Fluid Flow Solvers.

[11]  L. Čížková,et al.  Numerical Linear Algebra , 2018, Encyclopedia of Parallel Computing.

[12]  Miroslav Tuma,et al.  The Numerical Stability Analysis of Pipelined Conjugate Gradient Methods: Historical Context and Methodology , 2018, SIAM J. Sci. Comput..

[13]  Zdenek Strakos,et al.  Laplacian Preconditioning of Elliptic PDEs: Localization of the Eigenvalues of the Discretized Operator , 2018, SIAM J. Numer. Anal..

[14]  P. Joyce.  Matrices , 2018, Numerical Methods in Physics with Python.

[15]  Lloyd N. Trefethen,et al.  GMRES/CR and Arnoldi/Lanczos as Matrix Approximation Problems , 2018, SIAM J. Sci. Comput..

[16]  Iveta Hnetynková,et al.  Relating Computed and Exact Entities in Methods Based on Lanczos Tridiagonalization , 2017, HPCSE.

[17]  Emmanuel Agullo,et al.  Analyzing the Effect of Local Rounding Error Propagation on the Maximal Attainable Accuracy of the Pipelined Conjugate Gradient Method , 2016, SIAM J. Matrix Anal. Appl..

[18]  A. Greenbaum,et al.  Matrices That Generate the Same Krylov Residual Spaces , 2015 .

[19]  Anne Greenbaum,et al.  Predicting the Behavior of Finite Precision Lanczos and Conjugate Gradient Computations , 2015, SIAM J. Matrix Anal. Appl..

[20]  Zdenek Strakos,et al.  Preconditioning and the Conjugate Gradient Method in the Context of Solving PDEs , 2014, SIAM spotlights.

[21]  P. Deift,et al.  Universality in numerical computations with random data , 2014, Proceedings of the National Academy of Sciences.

[22]  Howard C. Elman,et al.  IFISS: A Computational Laboratory for Investigating Incompressible Flow Problems , 2014, SIAM Rev..

[23]  Jörg Liesen,et al.  Properties of Worst-Case GMRES , 2013, SIAM J. Matrix Anal. Appl..

[24]  J. Liesen,et al.  Max-min and min-max approximation problems for normal matrices revisited , 2013, 1310.5880.

[25]  Tomáš Gergelits,et al.  Analysis of Krylov subspace methods , 2013 .

[26]  Y. Saad,et al.  Approximating Spectral Densities of Large Matrices , 2013, SIAM Rev..

[27]  Zdenek Strakos,et al.  Model reduction using the Vorobyev moment problem , 2009, Numerical Algorithms.

[28]  G. Meurant.  The Lanczos and Conjugate Gradient Algorithms: From Theory to Finite Precision Computations , 2006 .

[29]  Miroslav Rozlozník,et al.  Modified Gram-Schmidt (MGS), Least Squares, and Backward Stability of MGS-GMRES , 2006, SIAM J. Matrix Anal. Appl..

[30]  G. Meurant,et al.  The Lanczos and conjugate gradient algorithms in finite precision arithmetic , 2006, Acta Numerica.

[31]  K. Mekchay.  Convergence of Adaptive Finite Element Methods , 2005 .

[32]  Z. Strakos,et al.  Error Estimation in Preconditioned Conjugate Gradients , 2005 .

[33]  Julien Langou,et al.  Rounding error analysis of the classical Gram-Schmidt orthogonalization process , 2005, Numerische Mathematik.

[34]  Zdenek Strakos,et al.  GMRES Convergence Analysis for a Convection-Diffusion Model Problem , 2005, SIAM J. Sci. Comput..

[35]  Jörg Liesen,et al.  Convergence analysis of Krylov subspace methods , 2004 .

[36]  Yousef Saad,et al.  Iterative methods for sparse linear systems , 2003 .

[37]  H. A. Vorst.  Iterative Krylov Methods for Large Linear Systems , 2003 .

[38]  Ricardo H. Nochetto,et al.  Convergence of Adaptive Finite Element Methods , 2002, SIAM Rev..

[39]  Christopher C. Paige,et al.  Scaled total least squares fundamentals , 2002, Numerische Mathematik.

[40]  Zdenek Strakos,et al.  Residual and Backward Error Bounds in Minimum Residual Krylov Subspace Methods , 2001, SIAM J. Sci. Comput..

[41]  M. Eiermann,et al.  Geometric aspects of the theory of Krylov subspace methods , 2001, Acta Numerica.

[42]  Zdenek Strakos,et al.  Accuracy of Two Three-term and Three Two-term Recurrences for Krylov Space Solvers , 2000, SIAM J. Matrix Anal. Appl..

[43]  Gene H. Golub,et al.  A Note on Preconditioning for Indefinite Linear Systems , 1999, SIAM J. Sci. Comput..

[44]  M. Arioli,et al.  Krylov sequences of maximal length and convergence of GMRES , 1997 .

[45]  G. Golub,et al.  Matrices, moments and quadrature II; How to compute the norm of the error in iterative methods , 1997 .

[46]  A. Edelman.  The Probability that a Random Real Gaussian Matrix haskReal Eigenvalues, Related Distributions, and the Circular Law , 1997 .

[47]  Anne Greenbaum,et al.  Any Nonincreasing Convergence Curve is Possible for GMRES , 1996, SIAM J. Matrix Anal. Appl..

[48]  Vladimir Druskin,et al.  Krylov subspace approximation of eigenpairs and matrix functions in exact and computer arithmetic , 1995, Numer. Linear Algebra Appl..

[49]  Gene H. Golub,et al.  Estimates in quadratic formulas , 1994, Numerical Algorithms.

[50]  Wayne Joubert,et al.  A Robust GMRES-Based Adaptive Polynomial Preconditioning Algorithm for Nonsymmetric Linear Systems , 1994, SIAM J. Sci. Comput..

[51]  Anne Greenbaum,et al.  Max-Min Properties of Matrix Factor Norms , 1994, SIAM J. Sci. Comput..

[52]  Henk A. van der Vorst,et al.  Bi-CGSTAB: A Fast and Smoothly Converging Variant of Bi-CG for the Solution of Nonsymmetric Linear Systems , 1992, SIAM J. Sci. Comput..

[53]  L. Trefethen,et al.  Eigenvalues and pseudo-eigenvalues of Toeplitz matrices , 1992 .

[54]  R. Freund,et al.  QMR: a quasi-minimal residual method for non-Hermitian linear systems , 1991 .

[55]  G. Golub,et al.  Iterative solution of linear systems , 1991, Acta Numerica.

[56]  Z. Strakos,et al.  On the real convergence rate of the conjugate gradient method , 1991 .

[57]  Gene H. Golub,et al.  Some History of the Conjugate Gradient and Lanczos Algorithms: 1948-1976 , 1989, SIAM Rev..

[58]  A. Greenbaum.  Behavior of slightly perturbed Lanczos and conjugate-gradient recurrences , 1989 .

[59]  Y. Saad,et al.  GMRES: a generalized minimal residual algorithm for solving nonsymmetric linear systems , 1986 .

[60]  H. V. D. Vorst,et al.  The rate of convergence of Conjugate Gradients , 1986 .

[61]  J. W. Silverstein.  The Smallest Eigenvalue of a Large Dimensional Wishart Matrix , 1985 .

[62]  S. Geman.  A Limit Theorem for the Norm of Random Matrices , 1980 .

[63]  A. Greenbaum.  Comparison of splittings used with the conjugate gradient algorithm , 1979 .

[64]  A. Jennings.  Influence of the Eigenvalue Spectrum on the Convergence Rate of the Conjugate Gradient Method , 1977 .

[65]  M. Saunders,et al.  Solution of Sparse Indefinite Systems of Linear Equations , 1975 .

[66]  M. Hestenes,et al.  Methods of conjugate gradients for solving linear systems , 1952 .

[67]  C. Lanczos.  Chebyshev polynomials in the solution of large-scale linear systems , 1952, ACM '52.

[68]  C. Lanczos.  Solution of Systems of Linear Equations by Minimized Iterations1 , 1952 .

[69]  W. Karush.  Convergence of a method of solving linear problems , 1952 .

[70]  W. Arnoldi.  The principle of minimized iterations in the solution of the matrix eigenvalue problem , 1951 .

[71]  Gérard Meurant,et al.  Krylov Methods for Nonsymmetric Linear Systems , 2020 .

[72]  Erin Carson,et al.  Communication-Avoiding Krylov Subspace Methods in Theory and Practice , 2015 .

[73]  W. Marsden.  I and J , 2012 .

[74]  G. Meurant.  The Lanczos and conjugate gradient algorithms , 2008 .

[75]  Numerische,et al.  On the convergence rate of the conjugate gradients in presence of rounding errors * , 2005 .

[76]  Z. Strakos,et al.  On error estimation in the conjugate gradient method and why it works in finite precision computations. , 2002 .

[77]  Arno B. J. Kuijlaars,et al.  Superlinear Convergence of Conjugate Gradients , 2001, SIAM J. Numer. Anal..

[78]  Arno B. J. Kuijlaars,et al.  On The Sharpness of an Asymptotic Error Estimate for Conjugate Gradients , 2001 .

[79]  Anne Greenbaum,et al.  Iterative methods for solving linear systems , 1997, Frontiers in applied mathematics.

[80]  Gene H. Golub,et al.  Closer to the solutions: iterative linear solvers , 1997 .

[81]  J. Kuczy,et al.  Estimating the Largest Eigenvalue by the Power and Lanczos Algorithms with a Random Start , 1992 .

[82]  Y. Saad.  Krylov subspace methods for solving large unsymmetric linear systems , 1981 .

[83]  P. Wesseling,et al.  Numerical experiments with a multiple grid and a preconditioned Lanczos type method , 1980 .

[84]  R. Fletcher.  Conjugate gradient methods for indefinite systems , 1976 .

[85]  I︠u︡. V. Vorobʹev.  Method of moments in applied mathematics , 1965 .

[86]  An iterative method for the solution of the eigenvalue problem of linear differential and integral , 1950 .