New Fast and Accurate Jacobi SVD Algorithm. I

This paper is the result of concerted efforts to break the barrier between numerical accuracy and run-time efficiency in computing the fundamental decomposition of numerical linear algebra—the singular value decomposition (SVD) of general dense matrices. It is an unfortunate fact that the numerically most accurate one-sided Jacobi SVD algorithm is several times slower than generally less accurate bidiagonalization-based methods such as the QR or the divide-and-conquer algorithm. Our quest for a highly accurate and efficient SVD algorithm has led us to a new, superior variant of the Jacobi algorithm. The new algorithm has inherited all good high accuracy properties of the Jacobi algorithm, and it can outperform the QR algorithm.

[1]  Alan George,et al.  Some Properties of Symmetric Quasi-Definite Matrices , 2000, SIAM J. Matrix Anal. Appl..

[2]  Josip Matejas,et al.  Quadratic convergence of scaled matrices in Jacobi method , 2000, Numerische Mathematik.

[3]  G. Golub,et al.  Linear least squares solutions by householder transformations , 1965 .

[4]  Ed Anderson,et al.  LAPACK Users' Guide , 1995 .

[5]  Stanley C. Eisenstat,et al.  A Divide-and-Conquer Algorithm for the Bidiagonal SVD , 1995, SIAM J. Matrix Anal. Appl..

[6]  James Demmel,et al.  Accurate Singular Values of Bidiagonal Matrices , 1990, SIAM J. Sci. Comput..

[7]  Beresford N. Parlett,et al.  Implicit Cholesky algorithms for singular values and vectors of triangular matrices , 1993, Numer. Linear Algebra Appl..

[8]  C. Jacobi,et al.  C. G. J. Jacobi's Gesammelte Werke: Über ein leichtes Verfahren, die in der Theorie der Sacularstorungen vorkommenden Gleichungen numerisch aufzulosen , 1846 .

[9]  Zvonimir Bujanovic,et al.  On the Failure of Rank-Revealing QR Factorization Software -- A Case Study , 2008, TOMS.

[10]  James Demmel,et al.  Cache efficient bidiagonalization using BLAS 2.5 operators , 2008, TOMS.

[11]  A. Ostrowski.  A QUANTITATIVE FORMULATION OF SYLVESTER'S LAW OF INERTIA. , 1959, Proceedings of the National Academy of Sciences of the United States of America.

[12]  James Demmel,et al.  Faster numerical algorithms via exception handling , 1993, Proceedings of IEEE 11th Symposium on Computer Arithmetic.

[13]  G. Stewart.  The Efficient Generation of Random Orthogonal Matrices with an Application to Condition Estimators , 1980 .

[14]  Ming Gu,et al.  Efficient Algorithms for Computing a Strong Rank-Revealing QR Factorization , 1996, SIAM J. Sci. Comput..

[15]  Charles L. Lawson,et al.  Solving least squares problems , 1976, Classics in applied mathematics.

[16]  W. Kahan.  Numerical Linear Algebra , 1966, Canadian Mathematical Bulletin.

[17]  Z. Drmač,et al.  On Scaled Almost-Diagonal Hermitian Matrix Pairs , 1997 .

[18]  V. Hari,et al.  On Jacobi methods for singular value decompositions , 1987 .

[19]  Zlatko Drmac,et al.  New Fast and Accurate Jacobi SVD Algorithm. II , 2007, SIAM J. Matrix Anal. Appl..

[20]  Zlatko Drmac,et al.  Implementation of Jacobi Rotations for Accurate Singular Value Computation in Floating Point Arithmetic , 1997, SIAM J. Sci. Comput..

[21]  M. Hestenes.  Inversion of Matrices by Biorthogonalization and Related Results , 1958 .

[22]  Herman H. Goldstine,et al.  The Jacobi Method for Real Symmetric Matrices , 1959, JACM.

[23]  Vjeran Hari,et al.  Convergence of a Block-Oriented Quasi-Cyclic Jacobi Method , 2007, SIAM J. Matrix Anal. Appl..

[24]  Nicholas J. Higham,et al.  INVERSE PROBLEMS NEWSLETTER , 1991 .

[25]  J. G. Lewis,et al.  Sparse Multifrontal Rank Revealing QR Factorization , 1997 .

[26]  J. Barlow,et al.  Computing accurate eigensystems of scaled diagonally dominant matrices: LAPACK working note No. 7 , 1988 .

[27]  G. Forsythe,et al.  The cyclic Jacobi method for computing the principal values of a complex matrix , 1960 .

[28]  Philip I. Davies,et al.  Numerically Stable Generation of Correlation Matrices and Their Factors , 2000 .

[29]  William Kahan,et al.  The baleful effect of computer benchmarks upon applied mathematics , 1997 .

[30]  J. Demmel,et al.  Computing the Singular Value Decomposition with High Relative Accuracy , 1997 .

[31]  V. Hari.  On sharp quadratic convergence bounds for the serial Jacobi methods , 1991 .

[32]  N. Rhee,et al.  On the global and cubic convergence of a quasi-cyclic Jacobi method , 1993 .

[33]  Ren-Cang Li,et al.  Relative Perturbation Theory: II. Eigenspace and Singular Subspace Variations , 1996, SIAM J. Matrix Anal. Appl..

[34]  C. Jacobi.  Ueber eine neue Auflösungsart der bei der Methode der kleinsten Quadrate vorkommenden lineären Gleichungen , 1845 .

[35]  V. Hari,et al.  Accelerating the SVD Block-Jacobi Method , 2005, Computing.

[36]  P. Tang,et al.  Bounds on singular values revealed by QR factorizations , 1995 .

[37]  Nicholas J. Higham,et al.  QR factorization with complete pivoting and accurate computation of the SVD , 2000 .

[38]  A. Sluis.  Condition numbers and equilibration of matrices , 1969 .

[39]  Zlatko Drmac,et al.  On Principal Angles between Subspaces of Euclidean Space , 2000, SIAM J. Matrix Anal. Appl..

[40]  Tony F. Chan,et al.  An Improved Algorithm for Computing the Singular Value Decomposition , 1982, TOMS.

[41]  B. Lang,et al.  An O(n2) algorithm for the bidiagonal SVD , 2003 .

[42]  Jesse L. Barlow,et al.  More Accurate Bidiagonal Reduction for Computing the Singular Value Decomposition , 2001, SIAM J. Matrix Anal. Appl..

[43]  John K. Reid,et al.  On applying Householder transformations to linear least squares problems , 1968, IFIP Congress.

[44]  B. Parlett,et al.  Accurate singular values and differential qd algorithms , 1994 .

[45]  B. Rust.  Truncating the Singular Value Decomposition for Ill-Posed Problems , 1998 .

[46]  T. Chan.  Rank Revealing OR Factorizations * , 2001 .

[47]  Ji-guang Sun.  Componentwise perturbation bounds for some matrix decompositions , 1992 .

[48]  G. W. Stewart,et al.  The QLP Approximation to the Singular Value Decomposition , 1999, SIAM J. Sci. Comput..

[49]  N. N. Chan,et al.  Diagonal elements and eigenvalues of a real symmetric matrix , 1983 .

[50]  J. H. Wilkinson.  Note on the quadratic convergence of the cyclic Jacobi process , 1962 .

[51]  Ilse C. F. Ipsen,et al.  On Rank-Revealing Factorisations , 1994, SIAM J. Matrix Anal. Appl..

[52]  G. Stewart.  Perturbation theory for the singular value decomposition , 1990 .

[53]  Jack J. Dongarra,et al.  A set of level 3 basic linear algebra subprograms , 1990, TOMS.

[54]  B. Parlett.  The Symmetric Eigenvalue Problem , 1981 .

[55]  Christian H. Bischof,et al.  Algorithm 782: codes for rank-revealing QR factorizations of dense matrices , 1998, TOMS.

[56]  James Demmel,et al.  Jacobi's Method is More Accurate than QR , 1989, SIAM J. Matrix Anal. Appl..

[57]  F. L. Bauer.  Software and Software Engineering , 1973 .

[58]  Ilse C. F. Ipsen.  Relative perturbation results for matrix eigenvalues and singular values , 1998, Acta Numerica.

[59]  P. Tang,et al.  Bounds on Singular Values Revealed by QR Factorizations , 1999 .

[60]  Inderjit S. Dhillon,et al.  Orthogonal Eigenvectors and Relative Gaps , 2003, SIAM J. Matrix Anal. Appl..

[61]  G. Golub,et al.  Eigenvalue computation in the 20th century , 2000 .

[62]  Z. Drmač.  A posteriori computation of the singular vectors in a preconditioned Jacobi SVD algorithm , 1999 .

[63]  Z. Drmač,et al.  On the Perturbation of the Cholesky Factorization , 1994 .

[64]  R. T. Gregory.  Computing eigenvalues and eigenvectors of a symmetric matrix on the ILLIAC , 1953 .

[65]  Ilse C. F. Ipsen,et al.  Analysis of a QR Algorithm for Computing Singular Values , 1995, SIAM J. Matrix Anal. Appl..

[66]  C. Jacobi.  Über ein leichtes Verfahren die in der Theorie der Säcularstörungen vorkommenden Gleichungen numerisch aufzulösen*). , 2022 .

[67]  Christian H. Bischof,et al.  Computing rank-revealing QR factorizations of dense matrices , 1998, TOMS.

[68]  W. Gentleman.  Error analysis of QR decompositions by Givens transformations , 1975 .

[69]  Ilse C. F. Ipsen,et al.  Relative perturbation techniques for singular value problems , 1995 .

[70]  P. P. Rijk.  A one-sided Jacobi algorithm for computing the singular value decomposition on avector computer , 1989 .

[71]  Walter F. Mascarenhas,et al.  On the Convergence of the Jacobi Method for Arbitrary Orderings , 1995, SIAM J. Matrix Anal. Appl..

[72]  A. Ostrowski,et al.  A QUANTITATIVE FORMULATION OF SYLVESTER'S LAW OF INERTIA, II. , 1959, Proceedings of the National Academy of Sciences of the United States of America.

[73]  Ren-Cang Li.  Relative Perturbation Theory: I. Eigenvalue and Singular Value Variations , 1998, SIAM J. Matrix Anal. Appl..

[74]  Christian H. Bischof,et al.  A BLAS-3 Version of the QR Factorization with Column Pivoting , 1998, SIAM J. Sci. Comput..

[75]  V. Hari,et al.  A note on a one-sided Jacobi algorithm , 1989 .

[76]  R. Hanson.  A Numerical Method for Solving Fredholm Integral Equations of the First Kind Using Singular Values , 1971 .

[77]  James Demmel.  Underflow and the Reliability of Numerical Software , 1984 .