Do Semidefinite Relaxations Really Solve Sparse PCA

Estimating the leading principal components of data assuming they are sparse, is a central task in modern high-dimensional statistics. Many algorithms were suggested for this sparse PCA problem, from simple diagonal thresholding to sophisticated semidefinite programming (SDP) methods. A key theoretical question asks under what conditions can such algorithms recover the sparse principal components. We study this question for a single-spike model, with a spike that is `0-sparse, and dimension p and sample size n that tend to infinity. Amini and Wainwright (2009) proved that for sparsity levels k ≥ Ω(n/ log p), no algorithm, efficient or not, can reliably recover the sparse eigenvector. In contrast, for sparsity levels k ≤ O( √ n/ log p), diagonal thresholding is asymptotically consistent.

[1]  R. Muirhead.  Aspects of Multivariate Statistical Theory , 1982, Wiley Series in Probability and Statistics.

[2]  Noga Alon,et al.  Finding a large hidden clique in a random graph , 1998, SODA '98.

[3]  U. Feige,et al.  Finding and certifying a large hidden clique in a semirandom graph , 2000 .

[4]  P. Massart,et al.  Adaptive estimation of a quadratic functional by model selection , 2000 .

[5]  I. Johnstone.  On the distribution of the largest eigenvalue in principal components analysis , 2001 .

[6]  Hongyuan Zha,et al.  Low-Rank Approximations with Sparse Factors I: Basic Algorithms and Error Analysis , 2001, SIAM J. Matrix Anal. Appl..

[7]  Olivier Ledoit,et al.  Some hypothesis tests for the covariance matrix when the dimension is large compared to the sample size , 2002 .

[8]  I. Jolliffe,et al.  A Modified Principal Component Technique Based on the LASSO , 2003 .

[9]  J. W. Silverstein,et al.  Eigenvalues of large sample covariance matrices of spiked population models , 2004, math/0408165.

[10]  Anja Vogler,et al.  An Introduction to Multivariate Statistical Analysis , 2004 .

[11]  Michael I. Jordan,et al.  A Direct Formulation for Sparse Pca Using Semidefinite Programming , 2004, NIPS 2004.

[12]  Shai Avidan,et al.  Spectral Bounds for Sparse PCA: Exact and Greedy Algorithms , 2005, NIPS.

[13]  R. Tibshirani,et al.  Sparse Principal Component Analysis , 2006 .

[14]  D. Paul.  ASYMPTOTICS OF SAMPLE EIGENSTRUCTURE FOR A LARGE DIMENSIONAL SPIKED COVARIANCE MODEL , 2007 .

[15]  P. Bickel,et al.  Regularized estimation of large covariance matrices , 2008, 0803.1909.

[16]  Jianhua Z. Huang,et al.  Sparse principal component analysis via regularized low rank matrix approximation , 2008 .

[17]  Alexandre d'Aspremont,et al.  First-Order Methods for Sparse Covariance Selection , 2006, SIAM J. Matrix Anal. Appl..

[18]  M. Wainwright,et al.  High-dimensional analysis of semidefinite relaxations for sparse principal components , 2008, 2008 IEEE International Symposium on Information Theory.

[19]  I. Johnstone,et al.  On Consistency and Sparsity for Principal Components Analysis in High Dimensions , 2009, Journal of the American Statistical Association.

[20]  U. Feige,et al.  Finding hidden cliques in linear time , 2009 .

[21]  R. Tibshirani,et al.  A penalized matrix decomposition, with applications to sparse principal components and canonical correlation analysis. , 2009, Biostatistics.

[22]  B. Nadler.  Finite sample approximation results for principal component analysis: a matrix perturbation approach , 2009, 0901.3245.

[23]  Stephen A. Vavasis,et al.  Nuclear norm minimization for the planted clique and biclique problems , 2009, Math. Program..

[24]  Ian T. Jolliffe,et al.  Principal Component Analysis , 2002, International Encyclopedia of Statistical Science.

[25]  Yong Zhang,et al.  An augmented Lagrangian approach for sparse principal component analysis , 2009, Mathematical Programming.

[26]  Jing Lei,et al.  Minimax Rates of Estimation for Sparse PCA in High Dimensions , 2012, AISTATS.

[27]  P. Rigollet,et al.  Optimal detection of sparse principal components in high dimension , 2012, 1202.5070.

[28]  Zongming Ma.  Sparse Principal Component Analysis and Iterative Thresholding , 2011, 1112.2432.

[29]  Philippe Rigollet,et al.  Complexity Theoretic Lower Bounds for Sparse Principal Component Detection , 2013, COLT.

[30]  Yuval Peres,et al.  Finding Hidden Cliques in Linear Time with High Probability , 2010, Combinatorics, Probability and Computing.

[31]  B. Nadler,et al.  MINIMAX BOUNDS FOR SPARSE PCA WITH NOISY HIGH-DIMENSIONAL DATA. , 2012, Annals of statistics.