Speeding up sum-of-squares for tensor decomposition and planted sparse vectors

We consider two problems that arise in machine learning applications: the problem of recovering a planted sparse vector in a random linear subspace and the problem of decomposing a random low-rank overcomplete 3-tensor. For both problems, the best known guarantees are based on the sum-of-squares method. We develop new algorithms inspired by analyses of the sum-of-squares method. Our algorithms achieve the same or similar guarantees as sum-of-squares for these problems but the running time is significantly faster. For the planted sparse vector problem, we give an algorithm with running time nearly linear in the input size that approximately recovers a planted sparse vector with up to constant relative

[1]  Prasad Raghavendra,et al.  Approximating CSPs with global cardinality constraints using SDP hierarchies , 2011, SODA.

[2]  Aditya Bhaskara,et al.  Smoothed analysis of tensor decompositions , 2013, STOC.

[3]  Anima Anandkumar,et al.  Analyzing Tensor Power Method Dynamics: Applications to Learning Overcomplete Latent Variable Models , 2014, ArXiv.

[4]  S. Janson.  Gaussian Hilbert Spaces , 1997 .

[5]  Jonathan Shi,et al.  Tensor principal component analysis via sum-of-square proofs , 2015, COLT.

[6]  Johan Håstad,et al.  Tensor Rank is NP-Complete , 1989, ICALP.

[7]  Tengyu Ma,et al.  Decomposing Overcomplete 3rd Order Tensors using Sum-of-Squares Algorithms , 2015, APPROX-RANDOM.

[8]  Jean B. Lasserre,et al.  Global Optimization with Polynomials and the Problem of Moments , 2000, SIAM J. Optim..

[9]  Yurii Nesterov,et al.  Squared Functional Systems and Optimization Problems , 2000 .

[10]  Anima Anandkumar,et al.  Learning Overcomplete Latent Variable Models through Tensor Methods , 2014, COLT.

[11]  K. Roberts,et al.  Thesis , 2002 .

[12]  Roman Vershynin,et al.  Introduction to the non-asymptotic analysis of random matrices , 2010, Compressed Sensing.

[13]  Rudolf Ahlswede,et al.  Strong converse for identification via quantum channels , 2000, IEEE Trans. Inf. Theory.

[14]  Huan Wang,et al.  Exact Recovery of Sparsely-Used Dictionaries , 2012, COLT.

[15]  Joseph T. Chang,et al.  Full reconstruction of Markov models on evolutionary trees: identifiability and consistency. , 1996, Mathematical biosciences.

[16]  Prasad Raghavendra,et al.  Rounding Semidefinite Programming Hierarchies via Global Correlation , 2011, 2011 IEEE 52nd Annual Symposium on Foundations of Computer Science.

[17]  Laurent Demanet,et al.  Recovering the Sparsest Element in a Subspace , 2013, 1310.1654.

[18]  Wenceslas Fernandez de la Vega,et al.  Linear programming relaxations of maxcut , 2007, SODA '07.

[19]  Anima Anandkumar,et al.  A tensor approach to learning mixed membership community models , 2013, J. Mach. Learn. Res..

[20]  Tamara G. Kolda,et al.  Tensor Decompositions and Applications , 2009, SIAM Rev..

[21]  David Steurer,et al.  Sum-of-squares proofs and the quest toward optimal algorithms , 2014, Electron. Colloquium Comput. Complex..

[22]  John Wright,et al.  Finding a Sparse Vector in a Subspace: Linear Sparsity Using Alternating Directions , 2014, IEEE Transactions on Information Theory.

[23]  Avi Wigderson,et al.  Sum-of-Squares Lower Bounds for Sparse PCA , 2015, NIPS.

[24]  Ankur Moitra,et al.  Tensor Prediction, Rademacher Complexity and Random 3-XOR , 2015, ArXiv.

[25]  Joel A. Tropp,et al.  User-Friendly Tail Bounds for Sums of Random Matrices , 2010, Found. Comput. Math..

[26]  V. Peña,et al.  Decoupling Inequalities for the Tail Probabilities of Multivariate $U$-Statistics , 1993, math/9309211.

[27]  Qingqing Huang,et al.  Learning Mixtures of Gaussians in High Dimensions , 2015, STOC.

[28]  Anima Anandkumar,et al.  A Spectral Algorithm for Latent Dirichlet Allocation , 2012, Algorithmica.

[29]  Michael I. Jordan,et al.  A Direct Formulation for Sparse Pca Using Semidefinite Programming , 2004, NIPS 2004.

[30]  Venkatesan Guruswami,et al.  Faster SDP Hierarchy Solvers for Local Rounding Algorithms , 2012, 2012 IEEE 53rd Annual Symposium on Foundations of Computer Science.

[31]  David Steurer,et al.  Rounding sum-of-squares relaxations , 2013, Electron. Colloquium Comput. Complex..

[32]  Anima Anandkumar,et al.  Tensor decompositions for learning latent variable models , 2012, J. Mach. Learn. Res..

[33]  Phong Q. Nguyen,et al.  Learning a Parallelepiped: Cryptanalysis of GGH and NTRU Signatures , 2009, Journal of Cryptology.

[34]  P. Massart,et al.  Adaptive estimation of a quadratic functional by model selection , 2000 .

[35]  Richard A. Harshman,et al.  Foundations of the PARAFAC procedure: Models and conditions for an "explanatory" multi-model factor analysis , 1970 .

[36]  François Le Gall,et al.  Faster Algorithms for Rectangular Matrix Multiplication , 2012, 2012 IEEE 53rd Annual Symposium on Foundations of Computer Science.

[37]  Michael I. Jordan,et al.  A Direct Formulation for Sparse Pca Using Semidefinite Programming , 2004, SIAM Rev..

[38]  Santosh S. Vempala,et al.  Fourier PCA and robust tensor decomposition , 2013, STOC.

[39]  Christopher J. Hillar,et al.  Most Tensor Problems Are NP-Hard , 2009, JACM.

[40]  Lieven De Lathauwer,et al.  Fourth-Order Cumulant-Based Blind Identification of Underdetermined Mixtures , 2007, IEEE Transactions on Signal Processing.

[41]  David Steurer,et al.  Dictionary Learning and Tensor Decomposition via the Sum-of-Squares Method , 2014, STOC.

[42]  Andrea Montanari,et al.  A statistical model for tensor PCA , 2014, NIPS.

[43]  Elchanan Mossel,et al.  Learning nonsingular phylogenies and hidden Markov models , 2005, STOC '05.

[44]  P. Parrilo.  Structured semidefinite programs and semialgebraic geometry methods in robustness and optimization , 2000 .

[45]  Avi Wigderson,et al.  Sum-of-squares Lower Bounds for Planted Clique , 2015, STOC.

[46]  Yuan Zhou,et al.  Hypercontractivity, sum-of-squares proofs, and their applications , 2012, STOC '12.