Low-Rank Matrix Approximations Do Not Need a Singular Value Gap

This is a systematic investigation into the sensitivity of low-rank approximations of real matrices. We show that the low-rank approximation errors, in the two-norm, Frobenius norm and more generally, any Schatten p-norm, are insensitive to additive rank-preserving perturbations in the projector basis; and to matrix perturbations that are additive or change the number of columns (including multiplicative perturbations). Thus, low-rank matrix approximations are always well-posed and do not require a singular value gap. In the presence of a singular value gap, connections are established between low-rank approximations and subspace angles.

[1]  C. Paige,et al.  History and generality of the CS decomposition , 1994 .

[2]  Ilse C. F. Ipsen,et al.  Randomized Approximation of the Gram Matrix: Exact Computation and Probabilistic Bounds , 2015, SIAM J. Matrix Anal. Appl..

[3]  Andrew V. Knyazev,et al.  Angles between subspaces and their tangents , 2012, J. Num. Math..

[4]  William Kahan,et al.  Some new bounds on perturbation of subspaces , 1969 .

[5]  Mei Han An,et al.  accuracy and stability of numerical algorithms , 1991 .

[6]  Petros Drineas,et al.  Fast Monte Carlo Algorithms for Matrices I: Approximating Matrix Multiplication , 2006, SIAM J. Comput..

[7]  Charles R. Johnson,et al.  Matrix analysis , 1985, Statistical Inference for Engineers and Data Scientists.

[8]  Abhisek Kundu,et al.  A Note on Randomized Element-wise Matrix Sparsification , 2014, ArXiv.

[9]  Siam J. Sci,et al.  SUBSPACE ITERATION RANDOMIZATION AND SINGULAR VALUE PROBLEMS , 2015 .

[10]  Cameron Musco,et al.  Randomized Block Krylov Methods for Stronger and Faster Approximate Singular Value Decomposition , 2015, NIPS.

[11]  Ilse C. F. Ipsen.  An overview of relative sin T theorems for invariant subspaces of complex matrices , 2000 .

[12]  G. Stewart,et al.  Matrix Perturbation Theory , 1990 .

[13]  P. Wedin.  Perturbation bounds in connection with singular value decomposition , 1972 .

[14]  David P. Woodruff.  Sketching as a Tool for Numerical Linear Algebra , 2014, Found. Trends Theor. Comput. Sci..

[15]  Y. Saad.  On the Rates of Convergence of the Lanczos and the Block-Lanczos Methods , 1980 .

[16]  Dimitris Achlioptas,et al.  Fast computation of low-rank matrix approximations , 2007, JACM.

[17]  Gene H. Golub,et al.  Matrix computations , 1983 .

[18]  G. W. Stewart,et al.  On the Numerical Analysis of Oblique Projectors , 2011, SIAM J. Matrix Anal. Appl..

[19]  Charles R. Johnson,et al.  Topics in Matrix Analysis , 1991 .

[20]  Petros Drineas,et al.  FAST MONTE CARLO ALGORITHMS FOR MATRICES II: COMPUTING A LOW-RANK APPROXIMATION TO A MATRIX∗ , 2004 .

[21]  G. Stewart.  Error and Perturbation Bounds for Subspaces Associated with Certain Eigenvalue Problems , 1973 .

[22]  W. Kahan,et al.  The Rotation of Eigenvectors by a Perturbation. III , 1970 .

[23]  Ilse C. F. Ipsen,et al.  Structural Convergence Results for Low-Rank Approximations from Block Krylov Spaces , 2016, ArXiv.

[24]  Chandler Davis.  The rotation of eigenvectors by a perturbation , 1963 .

[25]  Petros Drineas,et al.  A note on element-wise matrix sparsification via a matrix-valued Bernstein inequality , 2010, Inf. Process. Lett..