Structural Convergence Results for Low-Rank Approximations from Block Krylov Spaces

This paper is concerned with approximating the dominant left singular vector space of a real matrix $A$ of arbitrary dimension, from block Krylov spaces $\mathcal{K}_q(AA^T,AX)$. Two classes of results are presented. First are bounds on the distance, in the two and Frobenius norms, between the Krylov space and the target space. The distance is expressed in terms of principal angles. Second are quality of approximation bounds, relative to the best low-rank approximation in the Frobenius norm. For starting guesses $X$ of full column-rank, the bounds depend on the tangent of the principal angles between $X$ and the dominant right singular vector space of $A$. The results presented here form the structural foundation for the analysis of randomized Krylov space methods. The innovative feature is a combination of traditional Lanczos convergence analysis with optimal low-rank approximations via least squares problems.

[1]  David P. Woodruff,et al.  Low rank approximation and regression in input sparsity time , 2012, STOC '13.

[2]  Dimitris Achlioptas,et al.  Database-friendly random projections , 2001, PODS.

[3]  V. N. Bogaevski,et al.  Matrix Perturbation Theory , 1991 .

[4]  Danny C. Sorensen,et al.  Convergence of Polynomial Restart Krylov Methods for Eigenvalue Computations , 2005, SIAM Rev..

[5]  Robert H. Halstead,et al.  Matrix Computations , 2011, Encyclopedia of Parallel Computing.

[6]  Trac D. Tran,et al.  A fast and efficient algorithm for low-rank approximation of a matrix , 2009, STOC '09.

[7]  Michael W. Mahoney,et al.  Low-distortion subspace embeddings in input-sparsity time and applications to robust linear regression , 2012, STOC '13.

[8]  C. D. Meyer,et al.  Generalized inverses of linear transformations , 1979 .

[9]  Tamás Sarlós,et al.  Improved Approximation Algorithms for Large Matrices via Random Projections , 2006, 2006 47th Annual IEEE Symposium on Foundations of Computer Science (FOCS'06).

[10]  Philip Maher,et al.  Some norm inequalities concerning generalized inverses, 2 , 1992 .

[11]  Ren-Cang Li,et al.  Convergence of the block Lanczos method for eigenvalue clusters , 2014, Numerische Mathematik.

[12]  Charles R. Johnson,et al.  Topics in Matrix Analysis , 1991 .

[13]  Cameron Musco,et al.  Randomized Block Krylov Methods for Stronger and Faster Approximate Singular Value Decomposition , 2015, NIPS.

[14]  Y. Saad.  On the Rates of Convergence of the Lanczos and the Block-Lanczos Methods , 1980 .

[15]  Andrew V. Knyazev,et al.  Angles between subspaces and their tangents , 2012, J. Num. Math..

[16]  Bernard Chazelle,et al.  The Fast Johnson--Lindenstrauss Transform and Approximate Nearest Neighbors , 2009, SIAM J. Comput..

[17]  John Rossi,et al.  Convergence of Restarted Krylov Subspaces to Invariant Subspaces , 2004, SIAM J. Matrix Anal. Appl..

[18]  S. Muthukrishnan,et al.  Faster least squares approximation , 2007, Numerische Mathematik.

[19]  Zhihua Zhang,et al.  Improved Analyses of the Randomized Power Method and Block Lanczos Method , 2015, ArXiv.

[20]  Zlatko Drmač.  On relative residual bounds for the eigenvalues of a Hermitian matrix , 1996 .

[21]  Ming Gu,et al.  Subspace Iteration Randomization and Singular Value Problems , 2014, SIAM J. Sci. Comput..

[22]  Zlatko Drmac,et al.  Subspace Gap Residuals for Rayleigh-Ritz Approximations , 2009, SIAM J. Matrix Anal. Appl..

[23]  Joel A. Tropp,et al.  Improved Analysis of the subsampled Randomized Hadamard Transform , 2010, Adv. Data Sci. Adapt. Anal..

[24]  Petros Drineas,et al.  Structural Properties Underlying High-Quality Randomized Numerical Linear Algebra Algorithms , 2016, Handbook of Big Data.

[25]  D. Sorensen.  Numerical methods for large eigenvalue problems , 2002, Acta Numerica.

[26]  Avner Magen,et al.  Low rank matrix-valued chernoff bounds and approximate matrix multiplication , 2010, SODA '11.

[27]  Christos Boutsidis,et al.  Near Optimal Column-Based Matrix Reconstruction , 2011, 2011 IEEE 52nd Annual Symposium on Foundations of Computer Science.

[28]  Lothar Reichel,et al.  Restarted block Lanczos bidiagonalization methods , 2007, Numerical Algorithms.

[29]  Huy L. Nguyen,et al.  OSNAP: Faster Numerical Linear Algebra Algorithms via Sparser Subspace Embeddings , 2012, 2013 IEEE 54th Annual Symposium on Foundations of Computer Science.

[30]  S. Muthukrishnan,et al.  Relative-Error CUR Matrix Decompositions , 2007, SIAM J. Matrix Anal. Appl..

[31]  Å. Björck.  Numerical Methods in Matrix Computations , 2014 .

[32]  Lothar Reichel,et al.  Augmented Implicitly Restarted Lanczos Bidiagonalization Methods , 2005, SIAM J. Sci. Comput..