On the Approximability of Sparse PCA

It is well known that Sparse PCA (Sparse Principal Component Analysis) is NP-hard to solve exactly on worst-case instances. What is the complexity of solving Sparse PCA approximately? Our contributions include: 1. a simple and efficient algorithm that achieves ann 1=3 -approximation; 2. NP-hardness of approximation to within (1 "), for some small constant" > 0; 3. SSE-hardness of approximation to within any constant factor; and 4. an exp exp p log logn (“quasi-quasi-polynomial”) gap for the standard semidefinite program.

[1]  H. Kaiser.  The varimax criterion for analytic rotation in factor analysis , 1958 .

[2]  Yuan Zhou,et al.  Hypercontractivity, sum-of-squares proofs, and their applications , 2012, STOC '12.

[3]  Shai Avidan,et al.  Generalized spectral bounds for sparse LDA , 2006, ICML.

[4]  Subhash Khot,et al.  Improved inapproximability results for MaxClique, chromatic number and approximate graph coloring , 2001, Proceedings 2001 IEEE International Conference on Cluster Computing.

[5]  P. Rigollet,et al.  Optimal detection of sparse principal components in high dimension , 2012, 1202.5070.

[6]  Dimitris S. Papailiopoulos,et al.  Nonnegative Sparse PCA with Provable Guarantees , 2014, ICML.

[7]  T. Cai,et al.  Sparse PCA: Optimal rates and adaptive estimation , 2012, 1211.1309.

[8]  Alexandre d'Aspremont,et al.  Optimal Solutions for Sparse Principal Component Analysis , 2007, J. Mach. Learn. Res..

[9]  Yurii Nesterov,et al.  Generalized Power Method for Sparse Principal Component Analysis , 2008, J. Mach. Learn. Res..

[10]  Uriel Feige,et al.  The Dense k -Subgraph Problem , 2001, Algorithmica.

[11]  J. Håstad.  Clique is hard to approximate withinn1−ε , 1999 .

[12]  Noga Alon,et al.  The approximate rank of a matrix and its algorithmic applications: approximate rank , 2013, STOC '13.

[13]  I. Jolliffe,et al.  A Modified Principal Component Technique Based on the LASSO , 2003 .

[14]  Michael I. Jordan,et al.  A Direct Formulation for Sparse Pca Using Semidefinite Programming , 2004, SIAM Rev..

[15]  Lap Chi Lau,et al.  Lower Bounds on Expansions of Graph Powers , 2014, APPROX-RANDOM.

[16]  Subhash Khot,et al.  Ruling out PTAS for graph min-bisection, densest subgraph and bipartite clique , 2004, 45th Annual IEEE Symposium on Foundations of Computer Science.

[17]  Philippe Rigollet,et al.  Complexity Theoretic Lower Bounds for Sparse Principal Component Detection , 2013, COLT.

[18]  Aditya Bhaskara,et al.  Polynomial integrality gaps for strong SDP relaxations of Densest k-subgraph , 2011, SODA.

[19]  Madhu Sudan,et al.  Optimal Testing of Reed-Muller Codes , 2009, 2010 IEEE 51st Annual Symposium on Foundations of Computer Science.

[20]  Noga Alon,et al.  Testing Reed-Muller codes , 2005, IEEE Transactions on Information Theory.

[21]  Lap Chi Lau,et al.  Random Walks and Evolving Sets: Faster Convergences and Limitations , 2015, SODA.

[22]  Dimitris S. Papailiopoulos,et al.  Sparse PCA via Bipartite Matchings , 2015, NIPS.

[23]  Sanjeev Arora,et al.  Inapproximabilty of Densest κ-Subgraph from Average Case Hardness , 2011 .

[24]  M. Wainwright,et al.  High-dimensional analysis of semidefinite relaxations for sparse principal components , 2008, 2008 IEEE International Symposium on Information Theory.

[25]  Shai Avidan,et al.  Fast Pixel/Part Selection with Sparse Eigenvectors , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[26]  Mark Braverman,et al.  Approximating the best Nash Equilibrium in no(log n)-time breaks the Exponential Time Hypothesis , 2015, Electron. Colloquium Comput. Complex..

[27]  Prasad Raghavendra,et al.  Gap Amplification for Small-Set Expansion via Random Walks , 2014, APPROX-RANDOM.

[28]  Prasad Raghavendra,et al.  Reductions between Expansion Problems , 2010, 2012 IEEE 27th Conference on Computational Complexity.

[29]  Alexandre d'Aspremont,et al.  Approximation bounds for sparse principal component analysis , 2012, Math. Program..

[30]  I. Jolliffe.  Rotation of principal components: choice of normalization constraints , 1995 .

[31]  B. Nadler,et al.  DO SEMIDEFINITE RELAXATIONS SOLVE SPARSE PCA UP TO THE INFORMATION LIMIT , 2013, 1306.3690.

[32]  Omri Weinstein,et al.  ETH Hardness for Densest-k-Subgraph with Perfect Completeness , 2015, SODA.

[33]  Zongming Ma.  Sparse Principal Component Analysis and Iterative Thresholding , 2011, 1112.2432.

[34]  L. Ghaoui,et al.  Sparse PCA: Convex Relaxations, Algorithms and Applications , 2010, 1011.3781.

[35]  Gert R. G. Lanckriet,et al.  Sparse eigen methods by D.C. programming , 2007, ICML '07.

[36]  R. Tibshirani,et al.  Sparse Principal Component Analysis , 2006 .

[37]  Jianhua Z. Huang,et al.  Sparse principal component analysis via regularized low rank matrix approximation , 2008 .

[38]  Volodymyr Kuleshov,et al.  Fast algorithms for sparse principal component analysis based on Rayleigh quotient iteration , 2013, ICML.

[39]  Masafumi Yamashita,et al.  Approximating the longest path length of a stochastic DAG by a normal distribution in linear time , 2009, J. Discrete Algorithms.

[40]  David Zuckerman,et al.  Electronic Colloquium on Computational Complexity, Report No. 100 (2005) Linear Degree Extractors and the Inapproximability of MAX CLIQUE and CHROMATIC NUMBER , 2005 .

[41]  Xiao-Tong Yuan,et al.  Truncated power method for sparse eigenvalue problems , 2011, J. Mach. Learn. Res..

[42]  Malik Magdon-Ismail,et al.  NP-hardness and inapproximability of sparse PCA , 2015, Inf. Process. Lett..

[43]  Prasad Raghavendra,et al.  Graph expansion and the unique games conjecture , 2010, STOC '10.

[44]  Avi Wigderson,et al.  Sum-of-Squares Lower Bounds for Sparse PCA , 2015, NIPS.

[45]  Andrea Montanari,et al.  Sparse PCA via Covariance Thresholding , 2013, J. Mach. Learn. Res..

[46]  Harrison H. Zhou,et al.  Sparse CCA: Adaptive Estimation and Computational Barriers , 2014, 1409.8565.

[47]  Sanjeev Arora,et al.  Subexponential Algorithms for Unique Games and Related Problems , 2010, 2010 IEEE 51st Annual Symposium on Foundations of Computer Science.

[48]  Prasad Raghavendra,et al.  Making the Long Code Shorter , 2015, SIAM J. Comput..

[49]  Shai Avidan,et al.  Spectral Bounds for Sparse PCA: Exact and Greedy Algorithms , 2005, NIPS.

[50]  Aditya Bhaskara,et al.  Detecting high log-densities: an O(n¼) approximation for densest k-subgraph , 2010, STOC '10.

[51]  T. Cai,et al.  Optimal estimation and rank detection for sparse spiked covariance matrices , 2013, Probability theory and related fields.

[52]  Quentin Berthet,et al.  Statistical and computational trade-offs in estimation of sparse principal components , 2014, 1408.5369.

[53]  Prasad Raghavendra,et al.  Approximations for the isoperimetric and spectral profile of graphs and related parameters , 2010, STOC '10.

[54]  Yi Wu,et al.  Computational Complexity of Certifying Restricted Isometry Property , 2014, APPROX-RANDOM.

[55]  Johan Håstad,et al.  Clique is hard to approximate within n/sup 1-/spl epsiv// , 1996, Proceedings of 37th Conference on Foundations of Computer Science.

[56]  Jorge Cadima Departamento de Matematica.  Loading and correlations in the interpretation of principle compenents , 1995 .