Fast linear algebra is stable

In Demmel et al. (Numer. Math. 106(2), 199–224, 2007) we showed that a large class of fast recursive matrix multiplication algorithms is stable in a normwise sense, and that in fact if multiplication of n-by-n matrices can be done by any algorithm in O(nω+η) operations for any η >  0, then it can be done stably in O(nω+η) operations for any η >  0. Here we extend this result to show that essentially all standard linear algebra operations, including LU decomposition, QR decomposition, linear equation solving, matrix inversion, solving least squares problems, (generalized) eigenvalue problems and the singular value decomposition can also be done stably (in a normwise sense) in O(nω+η) operations.

[1]  V. Strassen.  Gaussian elimination is not optimal , 1969 .

[2]  K. Hadeler.  Submultiplikative Normen auf Algebren , 1969 .

[3]  R. Brent.  Algorithms for matrix multiplication , 1970 .

[4]  Allan Borodin,et al.  The computational complexity of algebraic and numeric problems , 1975, Elsevier computer science library.

[5]  D. Heller.  A Survey of Parallel Algorithms in Numerical Linear Algebra. , 1978 .

[6]  J. Varah.  On the Separation of Two Matrices , 1979 .

[7]  G. Stewart.  The Efficient Generation of Random Orthogonal Matrices with an Application to Condition Estimators , 1980 .

[8]  J. D. Roberts,et al.  Linear model reduction and solution of the algebraic Riccati equation by use of the sign function , 1980 .

[9]  Dario Bini,et al.  Stability of fast algorithms for matrix multiplication , 1980 .

[10]  R. Muirhead.  Aspects of Multivariate Statistical Theory , 1982, Wiley Series in Probability and Statistics.

[11]  Christian H. Bischof,et al.  The WY representation for products of householder matrices , 1985, PPSC.

[12]  S. Godunov.  Problem of the dichotomy of the spectrum of a matrix , 1986 .

[13]  Don Coppersmith,et al.  Matrix multiplication via arithmetic progressions , 1987, STOC.

[14]  T. W. Anderson,et al.  Generation of random orthogonal matrices , 1987 .

[15]  Jack Dongarra,et al.  ScaLAPACK Users' Guide , 1987 .

[16]  S. Godunov,et al.  Circular dichotomy of the spectrum of a matrix , 1988 .

[17]  A. Edelman.  Eigenvalues and condition numbers of random matrices , 1988 .

[18]  A. Malyshev.  Computing invariant subspaces of a regular linear pencil of matrices , 1989 .

[19]  C. Loan,et al.  A Storage-Efficient $WY$ Representation for Products of Householder Transformations , 1989 .

[20]  Volker Strassen,et al.  Algebraic Complexity Theory , 1991, Handbook of Theoretical Computer Science, Volume A: Algorithms and Complexity.

[21]  Nicholas J. Higham,et al.  Exploiting fast matrix multiplication within the level 3 BLAS , 1990, TOMS.

[22]  Nicholas J. Higham,et al.  INVERSE PROBLEMS NEWSLETTER , 1991 .

[23]  James Demmel,et al.  Stability of block algorithms with fast level-3 BLAS , 1992, TOMS.

[24]  C. Pan,et al.  Rank-Revealing QR Factorizations and the Singular Value Decomposition , 1992 .

[25]  P. Gács,et al.  Algorithms , 1992 .

[26]  A. Malyshev.  Parallel Algorithm for Solving Some Spectral Problems of Linear Algebra , 1993 .

[27]  Guodong Zhang,et al.  A Parallel Implementation of the Invariant Subspace Decomposition Algorithm for Dense Symmetric Matrices , 1993, PPSC.

[28]  David E. Keyes,et al.  Proceedings of the Sixth SIAM Conference on Parallel Processing for Scientific Computing, PPSC 1993, Norfolk, Virginia, USA, March 22-24, 1993 , 1993, PPSC.

[29]  James Demmel,et al.  Design of a Parallel Nonsymmetric Eigenroutine Toolbox, Part I , 1993, PPSC.

[30]  G. Stewart.  Updating a Rank-Revealing ULV Decomposition , 1993, SIAM J. Matrix Anal. Appl..

[31]  Xiaobai Sun,et al.  The PRISM project: infrastructure and algorithms for parallel eigensolvers , 1993, Proceedings of Scalable Parallel Libraries Conference.

[32]  Ilse C. F. Ipsen,et al.  On Rank-Revealing Factorisations , 1994, SIAM J. Matrix Anal. Appl..

[33]  James Demmel,et al.  Stability of block LU factorization , 1992, Numer. Linear Algebra Appl..

[34]  Nicholas J. Higham.  Stability of Parallel Triangular System Solvers , 1995, SIAM J. Sci. Comput..

[35]  Ming Gu,et al.  Efficient Algorithms for Computing a Strong Rank-Revealing QR Factorization , 1996, SIAM J. Sci. Comput..

[36]  Gene H. Golub,et al.  Matrix computations (3rd ed.) , 1996 .

[37]  D. Day.  How the QR algorithm fails to converge and how to fix it , 1996 .

[38]  Michael Clausen,et al.  Algebraic complexity theory , 1997, Grundlehren der mathematischen Wissenschaften.

[39]  James Demmel,et al.  Applied Numerical Linear Algebra , 1997 .

[40]  Sivan Toledo.  Locality of Reference in LU Decomposition with Partial Pivoting , 1997, SIAM J. Matrix Anal. Appl..

[41]  J. Demmel,et al.  An inverse free parallel spectral divide and conquer algorithm for nonsymmetric eigenproblems , 1997 .

[42]  Christian H. Bischof,et al.  Computing rank-revealing QR factorizations of dense matrices , 1998, TOMS.

[43]  J. Demmel,et al.  Using the Matrix Sign Function to Compute Invariant Subspaces , 1998, SIAM J. Matrix Anal. Appl..

[44]  James Demmel,et al.  LAPACK Users' Guide, Third Edition , 1999, Software, Environments and Tools.

[45]  Jack Dongarra,et al.  LAPACK Users' Guide, 3rd ed. , 1999 .

[46]  Erik Elmroth,et al.  Applying recursion to serial and parallel QR factorization leads to better performance , 2000, IBM J. Res. Dev..

[47]  Enrique S. Quintana-Ortí,et al.  Parallel Spectral Division Using the Matrix Sign Function for the Generalized Eigenproblem , 2000, Int. J. High Speed Comput..

[48]  Ran Raz,et al.  On the complexity of matrix product , 2002, STOC '02.

[49]  Christopher Umans,et al.  A group-theoretic approach to fast matrix multiplication , 2003, 44th Annual IEEE Symposium on Foundations of Computer Science, 2003. Proceedings..

[50]  Arnold Schönhage,et al.  Schnelle Multiplikation großer Zahlen , 1971, Computing.

[51]  Christopher Umans.  Group-theoretic algorithms for matrix multiplication , 2005, 46th Annual IEEE Symposium on Foundations of Computer Science (FOCS'05).

[52]  James Demmel,et al.  Fast matrix multiplication is stable , 2006, Numerische Mathematik.