Principal Component Analysis with Weighted Sparsity Constraint

Given a covariance matrix, principal component analysis (PCA) with sparsity constraint considers the problem of maximizing the variance explained by a particular linear combination of the input variables while constraining the number of nonzero coefficients in this combination. However, when loading an input variable is associated with an individual cost, we need to incorporate weights, which represent the loading cost of input variables, into sparsity constraint. And in this paper, we present a version of PCA with weighted sparsity constraint. This problem is reduced to solving some semidefinite programming ones via convex relaxation technique. Two applications of the PCA with weighted sparsity constraint to refine the sparsity constraint of sparse PCA illustrate its efficiency and reliability in practice.

[1]  H. Hotelling.  Analysis of a complex of statistical variables into principal components. , 1933 .

[2]  Jorge Cadima Departamento de Matematica.  Loading and correlations in the interpretation of principle compenents , 1995 .

[3]  S. Vines.  Simple principal components , 2000 .

[4]  R. Tibshirani,et al.  Sparse Principal Component Analysis , 2006 .

[5]  H. Sebastian Seung,et al.  Learning the parts of objects by non-negative matrix factorization , 1999, Nature.

[6]  Kim-Chuan Toh,et al.  SDPT3 -- A Matlab Software Package for Semidefinite Programming , 1996 .

[7]  Liviu Badea,et al.  Sparse Factorizations of Gene Expression Guided by Binding Data , 2005, Pacific Symposium on Biocomputing.

[8]  Alexander Schrijver,et al.  Cones of Matrices and Set-Functions and 0-1 Optimization , 1991, SIAM J. Optim..

[9]  Yurii Nesterov,et al.  Smoothing Technique and its Applications in Semidefinite Optimization , 2004, Math. Program..

[10]  Farid Alizadeh,et al.  Interior Point Methods in Semidefinite Programming with Applications to Combinatorial Optimization , 1995, SIAM J. Optim..

[11]  R. Jagannathan,et al.  Risk Reduction in Large Portfolios: Why Imposing the Wrong Constraints Helps , 2002 .

[12]  Shai Avidan,et al.  Spectral Bounds for Sparse PCA: Exact and Greedy Algorithms , 2005, NIPS.

[13]  C. Lemaréchal,et al.  Semidefinite Relaxations and Lagrangian Duality with Application to Combinatorial Optimization , 1999 .

[14]  J. N. R. Jeffers,et al.  Two Case Studies in the Application of Principal Component Analysis , 1967 .

[15]  Heng Tao Shen,et al.  Principal Component Analysis , 2009, Encyclopedia of Biometrics.

[16]  B. Moghaddam.  Spectral Bounds for Sparse PCA: Exact & Greedy Algorithms , .

[17]  Jos F. Sturm,et al.  A Matlab toolbox for optimization over symmetric cones , 1999 .

[18]  Danijel Skocaj,et al.  Weighted and robust incremental method for subspace learning , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[19]  Michael I. Jordan,et al.  A Direct Formulation for Sparse Pca Using Semidefinite Programming , 2004, SIAM Rev..

[20]  Sungyoung Lee,et al.  Two-dimensional weighted PCA algorithm for face recognition , 2005, 2005 International Symposium on Computational Intelligence in Robotics and Automation.

[21]  I. Jolliffe,et al.  A Modified Principal Component Technique Based on the LASSO , 2003 .

[22]  I. Jolliffe.  Rotation of principal components: choice of normalization constraints , 1995 .

[23]  Hongyuan Zha,et al.  Low-Rank Approximations with Sparse Factors II: Penalized Methods with Discrete Newton-Like Iterations , 2004, SIAM J. Matrix Anal. Appl..

[24]  Eugene Fiume,et al.  An efficient search algorithm for motion data using weighted PCA , 2005, SCA '05.

[25]  Hongyuan Zha,et al.  Low-Rank Approximations with Sparse Factors I: Basic Algorithms and Error Analysis , 2001, SIAM J. Matrix Anal. Appl..

[26]  Karl Pearson F.R.S..  LIII. On lines and planes of closest fit to systems of points in space , 1901 .

[27]  Liviu Badea,et al.  Sparse factorizations of gene expression data guided by binding data. , 2005, Pacific Symposium on Biocomputing. Pacific Symposium on Biocomputing.