An overview of block Gram-Schmidt methods and their stability properties

Block Gram-Schmidt algorithms comprise essential kernels in many scientific computing applications, but for many commonly used variants, a rigorous treatment of their stability properties remains open. This survey provides a comprehensive categorization of block Gram-Schmidt algorithms, especially those used in Krylov subspace methods to build orthonormal bases one block vector at a time. All known stability results are assembled, and new results are summarized or conjectured for important communication-reducing variants. A diverse array of numerical illustrations are presented, along with the MATLAB code for reproducing the results in a publicly available at repository this https URL. A number of open problems are discussed, and an appendix containing all algorithms type-set in a uniform fashion is provided.

[1]  Michael L. Parks,et al.  A block Recycled GMRES method with investigations into aspects of solver performance , 2016, ArXiv.

[2]  Jack J. Dongarra,et al.  Mixed-precision block gram Schmidt orthogonalization , 2015, ScalA '15.

[3]  G. Stewart,et al.  Reorthogonalization and stable algorithms for updating the Gram-Schmidt QR factorization , 1976 .

[4]  Erin C. Carson.  An Adaptive $s$-step Conjugate Gradient Algorithm with Dynamic Basis Updating , 2019, ArXiv.

[5]  Denis Vanderstraeten,et al.  An accurate parallel block Gram-Schmidt algorithm without reorthogonalization , 2000, Numerical Linear Algebra with Applications.

[6]  H. Walker.  Implementation of the GMRES method using householder transformations , 1988 .

[7]  Luke N. Olson,et al.  Node-Aware Improvements to Allreduce , 2019, 2019 IEEE/ACM Workshop on Exascale MPI (ExaMPI).

[8]  G. W. Stewart,et al.  Matrix Algorithms: Volume 1, Basic Decompositions , 1998 .

[9]  Stefano Zampini,et al.  KSPHPDDM and PCHPDDM: Extending PETSc with advanced Krylov methods and robust multilevel overlapping Schwarz preconditioners , 2021, Comput. Math. Appl..

[10]  W. Jalbyf,et al.  STABILITY ANALYSIS AND IMPROVEMENT OF THE BLOCK GRAM-SCHMIDT ALGORITHM , .

[11]  James Demmel,et al.  Communication lower bounds and optimal algorithms for numerical linear algebra*† , 2014, Acta Numerica.

[12]  B. Parlett.  The Symmetric Eigenvalue Problem , 1981 .

[13]  Yusaku Yamamoto,et al.  Roundoff error analysis of the Cholesky QR2 algorithm , 2015 .

[14]  Martin H. Gutknecht,et al.  Updating the QR decomposition of block tridiagonal and block Hessenberg matrices generated by block Krylov space methods , 2005 .

[15]  Daniel B. Szyld,et al.  Block Krylov Subspace Methods for Functions of Matrices II: Modified Block FOM , 2020, SIAM J. Matrix Anal. Appl..

[16]  D. O’Leary.  The block conjugate gradient algorithm and related methods , 1980 .

[17]  Miroslav Rozlozník,et al.  Modified Gram-Schmidt (MGS), Least Squares, and Backward Stability of MGS-GMRES , 2006, SIAM J. Matrix Anal. Appl..

[18]  L. Giraud,et al.  When modified Gram–Schmidt generates a well‐conditioned set of vectors , 2002 .

[19]  Christopher C. Paige,et al.  Loss and Recapture of Orthogonality in the Modified Gram-Schmidt Algorithm , 1992, SIAM J. Matrix Anal. Appl..

[20]  James Demmel,et al.  Communication-optimal Parallel and Sequential QR and LU Factorizations , 2008, SIAM J. Sci. Comput..

[21]  Laura Grigori,et al.  Enlarged Krylov Subspace Conjugate Gradient Methods for Reducing Communication , 2016, SIAM J. Matrix Anal. Appl..

[22]  Yousef Saad,et al.  Iterative methods for sparse linear systems , 2003 .

[23]  G. W. Stewart,et al.  A Krylov-Schur Algorithm for Large Eigenproblems , 2001, SIAM J. Matrix Anal. Appl..

[24]  Yusaku Yamamoto,et al.  Backward error analysis of the AllReduce algorithm for householder QR decomposition , 2011, Japan Journal of Industrial and Applied Mathematics.

[25]  Mark Hoemmen,et al.  Communication-avoiding Krylov subspace methods , 2010 .

[26]  W. Marsden.  I and J , 2012 .

[27]  Yusaku Yamamoto,et al.  Shifted Cholesky QR for Computing the QR Factorization of Ill-Conditioned Matrices , 2018, SIAM J. Sci. Comput..

[28]  T. Sakurai,et al.  A projection method for generalized eigenvalue problems using numerical integration , 2003 .

[29]  JESSE L. BARLOW,et al.  Block Modified Gram-Schmidt Algorithms and Their Analysis , 2019, SIAM J. Matrix Anal. Appl..

[30]  R. Morgan.  Restarted block-GMRES with deflation of eigenvalues , 2005 .

[31]  Nicholas J. Higham,et al.  INVERSE PROBLEMS NEWSLETTER , 1991 .

[32]  Shreyas Ananthan,et al.  Low synchronization Gram–Schmidt and generalized minimal residual algorithms , 2020, Numer. Linear Algebra Appl..

[33]  Xiaobai Sun,et al.  Aggregations of Elementary Transformations , 1996 .

[34]  Julien Langou,et al.  Rounding error analysis of the classical Gram-Schmidt orthogonalization process , 2005, Numerische Mathematik.

[35]  C. Loan,et al.  A Storage-Efficient $WY$ Representation for Products of Householder Transformations , 1989 .

[36]  B. Vital.  Etude de quelques methodes de resolution de problemes lineaires de grande taille sur multiprocesseur , 1990 .

[37]  L. Minah Yang,et al.  Mixed-Precision analysis of Householder QR Algorithms , 2019, ArXiv.

[38]  Axel Ruhe.  Numerical aspects of gram-schmidt orthogonalization of vectors , 1983 .

[39]  Eric de Sturler,et al.  Recycling Krylov Subspaces for Sequences of Linear Systems , 2006, SIAM J. Sci. Comput..

[40]  Jesse L. Barlow,et al.  Reorthogonalized block classical Gram–Schmidt , 2011, Numerische Mathematik.

[41]  M. Gutknecht.  BLOCK KRYLOV SPACE METHODS FOR LINEAR SYSTEMS WITH MULTIPLE RIGHT-HAND SIDES : AN , 2005 .

[42]  Jack J. Dongarra,et al.  Mixed-Precision Cholesky QR Factorization and Its Case Studies on Multicore CPU with Multiple GPUs , 2015, SIAM J. Sci. Comput..

[43]  Anthony T. Chronopoulos,et al.  An Efficient Parallel Algorithm for Extreme Eigenvalues of Sparse Nonsymmetric Matrices , 1992, Int. J. High Perform. Comput. Appl..

[44]  Stefan Güttel,et al.  Convergence of Restarted Krylov Subspace Methods for Stieltjes Functions of Matrices , 2014, SIAM J. Matrix Anal. Appl..

[45]  Walter Gander,et al.  Algorithms for the QR-Decomposition , 2003 .

[46]  Julien Langou,et al.  A note on the error analysis of classical Gram–Schmidt , 2006, Numerische Mathematik.

[47]  Erin Carson,et al.  Communication-Avoiding Krylov Subspace Methods in Theory and Practice , 2015 .

[48]  G. W. Stewart.  Block Gram--Schmidt Orthogonalization , 2008, SIAM J. Sci. Comput..

[49]  Walter Hoffmann,et al.  Iterative algorithms for Gram-Schmidt orthogonalization , 1989, Computing.

[50]  N. Abdelmalek.  Round off error analysis for Gram-Schmidt method and solution of linear least squares problems , 1971 .

[51]  V. Simoncini,et al.  Convergence properties of block GMRES and matrix polynomials , 1996 .

[52]  Andrew V. Knyazev,et al.  Toward the Optimal Preconditioned Eigensolver: Locally Optimal Block Preconditioned Conjugate Gradient Method , 2001, SIAM J. Sci. Comput..

[53]  Stefan Güttel,et al.  Efficient and Stable Arnoldi Restarts for Matrix Functions Based on Quadrature , 2014, SIAM J. Matrix Anal. Appl..

[54]  Stefan Güttel,et al.  Deflated Restarting for Matrix Functions , 2011, SIAM J. Matrix Anal. Appl..

[55]  S. Güttel.  Rational Krylov approximation of matrix functions: Numerical methods and optimal pole selection , 2013 .

[56]  Ichitaro Yamazaki,et al.  Low-synchronization orthogonalization schemes for s-step and pipelined Krylov solvers in Trilinos , 2020, PPSC.

[57]  Elizabeth R. Jessup,et al.  On Improving Linear Solver Performance: A Block Variant of GMRES , 2005, SIAM J. Sci. Comput..

[58]  Walter Gander,et al.  Gram‐Schmidt orthogonalization: 100 years and more , 2013, Numer. Linear Algebra Appl..

[59]  Å. Björck.  Solving linear least squares problems by Gram-Schmidt orthogonalization , 1967 .

[60]  Gene H. Golub,et al.  The block Lanczos method for computing eigenvalues , 2007, Milestones in Matrix Computation.

[61]  M. Sadkane.  Block-Arnoldi and Davidson methods for unsymmetric large eigenvalue problems , 1993 .