Low rank approximation and regression in input sparsity time

We design a new distribution over poly(r ε^-1) x n matrices S so that for any fixed n x d matrix A of rank r, with probability at least 9/10, SAx₂ = (1 pm ε)Ax₂ simultaneously for all x ∈ R^d. Such a matrix S is called a <i>subspace embedding</i>. Furthermore, SA can be computed in O(nnz(A)) + ~O(r²ε^-2) time, where nnz(A) is the number of non-zero entries of A. This improves over all previous subspace embeddings, which required at least Ω(nd log d) time to achieve this property. We call our matrices S <i>sparse embedding matrices</i>. Using our sparse embedding matrices, we obtain the fastest known algorithms for overconstrained least-squares regression, low-rank approximation, approximating all leverage scores, and l_p-regression: to output an x' for which Ax'-b₂ ≤ (1+ε)min_x Ax-b₂ for an n x d matrix A and an n x 1 column vector b, we obtain an algorithm running in O(nnz(A)) + ~O(d³ε^-2) time, and another in O(nnz(A)log(1/ε)) + ~O(d³log(1/ε)) time. (Here ~O(f) = f ⋅ log^O(1)(f).) to obtain a decomposition of an n x n matrix A into a product of an n x k matrix L, a k x k diagonal matrix D, and a n x k matrix W, for which F{A - L D W} ≤ (1+ε)F{A-A_k}, where A_k is the best rank-k approximation, our algorithm runs in O(nnz(A)) + ~O(nk² ε^-4log n + k³ε^-5log²n) time. to output an approximation to all leverage scores of an n x d input matrix A simultaneously, with constant relative error, our algorithms run in O(nnz(A) log n) + ~O(r³) time. to output an x' for which Ax'-b_p ≤ (1+ε)min_x Ax-b_p for an n x d matrix A and an n x 1 column vector b, we obtain an algorithm running in O(nnz(A) log n) + poly(r ε^-1) time, for any constant 1 ≤ p < ∞. We optimize the polynomial factors in the above stated running times, and show various tradeoffs. Finally, we provide preliminary experimental results which suggest that our algorithms are of interest in practice.