Matching Matrix Bernstein with Little Memory: Near-Optimal Finite Sample Guarantees for Oja's Algorithm

This work provides improved guarantees for streaming princ iple component analysis (PCA). Given A1, . . . ,An ∈ R sampled independently from distributions satisfying E [Ai] = Σ for Σ 0, this work provides anO(d)-space linear-time single-pass streaming algorithm for es timating the top eigenvector ofΣ. The algorithm nearly matches (and in certain cases improve s upon) the accuracy obtained by the standard batch method that computes top eigenvector o f he empirical covariance n ∑ i∈[n] Ai as analyzed by the matrix Bernstein inequality. Moreover, t o achieve constant accuracy, our algorithm improves upon the best previous known sample complexities o f streaming algorithms by either a multiplicative factor ofO(d) or 1/gap wheregap is the relative distance between the top two eigenvalues of Σ. These results are achieved through a novel analysis of the cl assi Oja’s algorithm, one of the oldest and most popular algorithms for streaming PCA. In particula r, this work shows that simply picking a random initial pointw0 and applying the update rule wi+1 = wi + ηiAiwi suffices to accurately estimate the top eigenvector, with a suitable choice of ηi. We believe our result sheds light on how to efficiently perform streaming PCA both in theory and in pract ice, and we hope that our analysis may serve as the basis for analyzing many variants and extension s of streaming PCA. Microsoft Research India. Email: prajain@microsoft.com UC Berkeley. Email: chijin@cs.berkeley.edu University of Washington. Email: sham@cs.washington.edu Microsoft Research New England. Email: praneeth@microsof t.com Microsoft Research New England. Email: asid@microsoft.co m