Trading Off Parallelism and Numerical Stability

The fastest parallel algorithm for a problem may be significantly less stable numerically than the fastest serial algorithm. We illustrate this phenomenon by a series of examples drawn from numerical linear algebra. We also show how some of these instabilities may be mitigated by better floating point arithmetic.

[1]  P. J. Eberlein.  Errata: A Jacobi-Like Method for the Automatic Computation of Eigenvalues and Eigenvectors , 1962 .

[2]  R. Brent.  Algorithms for matrix multiplication , 1970 .

[3]  T. J. Dekker,et al.  A floating-point technique for extending the available precision , 1971 .

[4]  R. Brent.  Error analysis of algorithms for matrix multiplication and triangular decomposition using Winograd's identity , 1970 .

[5]  A. Sameh.  On Jacobi and Jacobi-I ike Algorithms for a Parallel Computer , 2010 .

[6]  Alfred V. Aho,et al.  The Design and Analysis of Computer Algorithms , 1974 .

[7]  L. Csanky,et al.  Fast parallel matrix inversion algorithms , 1975, 16th Annual Symposium on Foundations of Computer Science (sfcs 1975).

[8]  L. Csanky,et al.  Fast Parallel Matrix Inversion Algorithms , 1976, SIAM J. Comput..

[9]  R. Brent,et al.  Solving Triangular Systems on a Parallel Computer , 1977 .

[10]  David J. Kuck,et al.  A Parallel QR Algorithm for Symmetric Tridiagonal Matrices , 1977, IEEE Transactions on Computers.

[11]  D Parkinson,et al.  High–Speed Computing , 1978 .

[12]  K. Veselic,et al.  A quadratically convergent Jacobi-like method for real matrices with complex eigenvalues , 1979 .

[13]  Charles L. Lawson,et al.  Basic Linear Algebra Subprograms for Fortran Usage , 1979, TOMS.

[14]  J. Cuppen.  A divide and conquer method for the symmetric tridiagonal eigenproblem , 1980 .

[15]  P. Bjørstad,et al.  Numerical solution of the biharmonic equation , 1980 .

[16]  Dario Bini,et al.  Stability of fast algorithms for matrix multiplication , 1980 .

[17]  Gene H. Golub,et al.  Matrix computations , 1983 .

[18]  G. Stewart.  A Jacobi-Like Algorithm for Computing the Schur Decomposition of a Nonhermitian Matrix , 1985 .

[19]  Jack J. Dongarra,et al.  A fully parallel algorithm for the symmetric eigenvalue problem , 1985, PPSC.

[20]  Danny C. Sorensen,et al.  Analysis of Pairwise Pivoting in Gaussian Elimination , 1985, IEEE Transactions on Computers.

[21]  R. Schreiber.  Solving Eigenvalue and Singular Value Problems on an Undersized Systolic Array , 1986 .

[22]  Ed Anderson,et al.  LAPACK Users' Guide , 1995 .

[23]  G. W. Stewart,et al.  A parallel implementation of the QR-algorithm , 1987, Parallel Comput..

[24]  Robert A. van de Geijn.  Implementing the qr-algorithm on an array of processors , 1987 .

[25]  Elizabeth R. Jessup,et al.  A Divide and Conquer Algorithm for Computing the Singular Value Decomposition , 1987, SIAM Conference on Parallel Processing for Scientific Computing.

[26]  Michael T. Heath.  Hypercube multiprocessors 1987 , 1987 .

[27]  J. Demmel.  On condition numbers and the distance to the nearest ill-posed problem , 2015 .

[28]  Patricia J. Eberlein,et al.  On the Schur Decomposition of a Matrix for Parallel Computation , 1985, IEEE Transactions on Computers.

[29]  A. Edelman.  Eigenvalues and condition numbers of random matrices , 1988 .

[30]  Martin H. Schultz,et al.  Numerical Algorithms for Modern Parallel Computer Architectures , 1988 .

[31]  Tien-Yien Li,et al.  Homotopy method for general l-matrix problems , 1988 .

[32]  Robert Schreiber,et al.  Block Algorithms for Parallel Machines , 1988 .

[33]  David H. Bailey,et al.  Extra high speed matrix multiplication on the Cray-2 , 1988 .

[34]  M. Chu.  A note on the homotopy method for linear algebraic eigenvalue problems , 1988 .

[35]  Al Geist.  Parallel Tridiagonalization of a General Matrix Using Distributed-Memory Multiprocessors , 1989, PPSC.

[36]  James Demmel,et al.  On a Block Implementation of Hessenberg Multishift QR Iteration , 1989, Int. J. High Speed Comput..

[37]  R. Schreiber,et al.  On the convergence of the cyclic Jacobi method for parallel block orderings , 1989 .

[38]  George A. Geist.  Reduction of a general matrix to tridiagonal form using a hypercube multiprocessor , 1991 .

[39]  M. Paardekooper.  A quadratically convergent parallel Jacobi process for diagonally dominant matrices with distinct eigenvalues , 1989 .

[40]  Jack J. Dongarra,et al.  A set of level 3 basic linear algebra subprograms , 1990, TOMS.

[41]  Jack Dongarra,et al.  LAPACK Working Note 26: Prospectus for an Extension to LAPACK: A Portable Linear Algebra Library for High-Performance Computers , 1990 .

[42]  Gautam M. Shroff.  A parallel algorithm for the eigenvalues and eigenvectors of a general complex matrix , 1990 .

[43]  M. Paardekooper,et al.  A quadratically convergent parallel Jacobi process for diagonally dominant matrices with distinct eigenvalues , 1990 .

[44]  L. Trefethen,et al.  Average-case stability of Gaussian elimination , 1990 .

[45]  Jack Dongarra,et al.  Computing the eigenvalues and eigenvectors of a general matrix by reduction to general tridiagonal form , 1990 .

[46]  Al Geist,et al.  Finding eigenvalues and eigenvectors of unsymmetric matrices using a distributed-memory multiprocessor , 1990, Parallel Comput..

[47]  K. A. Gallivan,et al.  Parallel Algorithms for Dense Linear Algebra Computations , 1990, SIAM Rev..

[48]  James Demmel,et al.  Accurate Singular Values of Bidiagonal Matrices , 1990, SIAM J. Sci. Comput..

[49]  Nicholas J. Higham,et al.  Exploiting fast matrix multiplication within the level 3 BLAS , 1990, TOMS.

[50]  D. Sorensen,et al.  Block reduction of matrices to condensed forms for eigenvalue computations , 1990 .

[51]  Ilse C. F. Ipsen,et al.  Solving the Symmetric Tridiagonal Eigenvalue Problem on the Hypercube , 1990, SIAM J. Sci. Comput..

[52]  Edward Anderson.  Robust Triangular Solves for Use in Condition Estimation , 1991 .

[53]  J. Dongarra,et al.  A Parallel Algorithm for the Non-Symmetric Eigenvalue Problem , 1991 .

[54]  D. Sorensen,et al.  On the orthogonality of eigenvectors computed by divide-and-conquer techniques , 1991 .

[55]  David S. Watkins,et al.  Convergence of algorithms of decomposition type for the eigenvalue problem , 1991 .

[56]  Douglas M. Priest,et al.  Algorithms for arbitrary precision floating point arithmetic , 1991, [1991] Proceedings 10th IEEE Symposium on Computer Arithmetic.

[57]  A. N. Malychev.  Parallel aspects of some spectral problems in linear algebra , 1991 .

[58]  Jessup.  A divide and conquer approach to the nonsymmetric eigenvalue problem , 1991 .

[59]  Ilse C. F. Ipsen,et al.  Improving the Accuracy of Inverse Iteration , 1992, SIAM J. Sci. Comput..

[60]  T. Y. Li,et al.  Solving eigenvalue problems of real nonsymmetric matrices with real homotopies , 1992 .

[61]  James Demmel,et al.  Stability of block algorithms with fast level-3 BLAS , 1992, TOMS.

[62]  James Demmel.  The inherent inaccuracy of implicit tridiagonal QR , 1992 .

[63]  Tien-Yien Li,et al.  Homotopy-determinant algorithm for solving nonsymmetric eigenvalue problems , 1992 .

[64]  Beresford N. Parlett,et al.  Reduction to Tridiagonal Form and Minimal Realizations , 1992, SIAM J. Matrix Anal. Appl..

[65]  James Demmel,et al.  Jacobi's Method is More Accurate than QR , 1989, SIAM J. Matrix Anal. Appl..

[66]  James Demmel,et al.  Block LU factorization , 1992 .

[67]  A. Edelman.  On the distribution of a scaled condition number , 1992 .

[68]  J. Barlow.  Error analysis of update methods for the symmetric eigenvalue problem , 1993 .

[69]  James Demmel,et al.  Parallel numerical linear algebra , 1993, Acta Numerica.

[70]  Ivan Slapničar,et al.  Accurate Symmetric Eigenreduction by a Jacobi Method , 1993 .

[71]  James Demmel,et al.  Design of a Parallel Nonsymmetric Eigenroutine Toolbox, Part I , 1993, PPSC.

[72]  P. Swarztrauber.  A parallel algorithm for computing the eigenvalues of a symmetric tridiagonal matrix , 1993 .

[73]  B. Parlett,et al.  Accurate singular values and differential qd algorithms , 1994 .

[74]  David S. Watkins,et al.  Shifting Strategies for the Parallel QR Algorithm , 1994, SIAM J. Sci. Comput..

[75]  B. Parlett,et al.  Accurate singular values and differential qd algorithms , 1994 .