Follow the Compressed Leader: Faster Algorithms for Matrix Multiplicative Weight Updates

Matrix multiplicative weight update (MMWU) is an extremely powerful algorithmic tool for computer science and related fields. However, it comes with a slow running time due to the matrix exponential and eigendecomposition computations. For this reason, many researchers studied the followed-the-perturbed-leader (FTPL) framework which is faster, but a factor $\sqrt{d}$ worse than the optimal regret of MMWU for dimension-$d$ matrices. In this paper, we propose a $\textit{followed-the-compressed-leader}$ framework which, not only matches the optimal regret of MMWU (up to polylog factors), but runs $\textit{even faster}$ than FTPL. Our main idea is to "compress" the matrix exponential computation to dimension 3 in the adversarial setting, or dimension 1 in the stochastic setting. This result resolves an open question regarding how to obtain both (nearly) optimal and efficient algorithms for the online eigenvector problem.

[1]  Jacob Abernethy,et al.  Spectral Smoothing via Random Matrix Perturbations , 2015, ArXiv.

[2]  Yin Tat Lee,et al.  Constructing Linear-Sized Spectral Sparsification in Almost-Linear Time , 2015, 2015 IEEE 56th Annual Symposium on Foundations of Computer Science.

[3]  Nisheeth K. Vishnoi,et al.  Approximating the exponential, the lanczos method and an Õ(m)-time spectral algorithm for balanced separator , 2011, STOC '12.

[4]  Rahul Jain,et al.  QIP = PSPACE , 2011, JACM.

[5]  J. Shewchuk.  An Introduction to the Conjugate Gradient Method Without the Agonizing Pain , 1994 .

[6]  Sanjeev Arora,et al.  A combinatorial, primal-dual approach to semidefinite programs , 2007, STOC.

[7]  Sanjeev Arora,et al.  The Multiplicative Weights Update Method: a Meta-Algorithm and Applications , 2012, Theory Comput..

[8]  Shai Shalev-Shwartz,et al.  SDCA without Duality, Regularization, and Individual Convexity , 2016, ICML.

[9]  Yin Tat Lee,et al.  Using Optimization to Obtain a Width-Independent, Parallel, Simpler, and Faster Positive SDP Solver , 2015, SODA.

[10]  Tengyu Ma,et al.  Online Learning of Eigenvectors , 2015, ICML.

[11]  Gábor Lugosi,et al.  Prediction, learning, and games , 2006 .

[12]  Zaïd Harchaoui,et al.  A Universal Catalyst for First-Order Optimization , 2015, NIPS.

[13]  Li Zhang,et al.  Analyze gauss: optimal bounds for privacy-preserving principal component analysis , 2014, STOC.

[14]  Yuanzhi Li,et al.  Even Faster SVD Decomposition Yet Without Agonizing Pain , 2016, NIPS.

[15]  G. Birkhoff.  Note on the gamma function , 1913 .

[16]  Ohad Shamir,et al.  Convergence of Stochastic Gradient Descent for PCA , 2015, ICML.

[17]  Lorenzo Orecchia,et al.  Fast Approximation Algorithms for Graph Partitioning Using Spectral and Semidefinite-Programming Techniques , 2011 .

[18]  Moritz Hardt,et al.  The Noisy Power Method: A Meta Algorithm with Applications , 2013, NIPS.

[19]  Zohar S. Karnin,et al.  Online {PCA} with Spectral Bounds , 2015 .

[20]  Wojciech Kotlowski,et al.  PCA with Gaussian perturbations , 2015, ArXiv.

[21]  Christos Boutsidis,et al.  Online Principal Components Analysis , 2015, SODA.

[22]  Zeyuan Allen Zhu,et al.  Improved SVRG for Non-Strongly-Convex or Sum-of-Non-Convex Objectives , 2015, ICML.

[23]  Yuanzhi Li,et al.  First Efficient Convergence for Streaming k-PCA: A Global, Gap-Free, and Near-Optimal Rate , 2016, 2017 IEEE 58th Annual Symposium on Foundations of Computer Science (FOCS).

[24]  Jiazhong Nie,et al.  Online PCA with Optimal Regrets , 2013, ALT.

[25]  Richard Peng,et al.  Faster and simpler width-independent parallel algorithms for positive semidefinite programming , 2012, SPAA '12.

[26]  Sham M. Kakade,et al.  Un-regularizing: approximate proximal point and faster stochastic algorithms for empirical risk minimization , 2015, ICML.

[27]  Ambuj Tewari,et al.  Online Linear Optimization via Smoothing , 2014, COLT.

[28]  Yuanzhi Li,et al.  Doubly Accelerated Methods for Faster CCA and Generalized Eigendecomposition , 2016, ICML.

[29]  Elad Hazan,et al.  Fast and Simple PCA via Convex Optimization , 2015, ArXiv.