Stronger and Faster Approximate Singular Value Decomposition via the Block Lanczos Method

Since being analyzed by Rokhlin, Szlam, and Tygert [1] and po pularized by Halko, Martinsson, and Tropp [2], randomized Simultaneous Power Iteration has become the method of choice for approximate singular value d ecomposition. It is more accurate than simpler sketching algorithms, yet still converges quickly for anymatrix, independently of singular value gaps. After Õ(1/ǫ) iterations, it gives a low-rank approximation within(1 + ǫ) of optimal for spectral norm error. We give the first provable runtime improvement on Simultaneo us Iteration – a simple randomized variant of the classic Block Lanczos meth od gives the same guarantees in just̃ O(1/ √ ǫ) iterations and performs substantially better experimentally. Despite their long history, our analysis is the fir st of a Krylov subspace method like Block Lanczos that does not depend on singular va lue gaps, which are unreliable in practice. Furthermore, while it is a simple accuracy benchmark, even (1 + ǫ) error for spectral norm low rank approximation does not imply that an a lgorithm returns high quality principal components, a major issue for data ap plications. We address this problem for the first time by showing that both Block Lanc zos and a minor modification of Simultaneous Iteration give nearly optimal PCA for any matrix. This result further justifies their strength over non-itera ive sketching methods. Finally, we give insight beyond the worst case, justifying w hy both algorithms can run much faster in practice than predicted. We clarify how si mple techniques can take advantage of common matrix properties to significantly improve runtime.