Multitask Principal Component Analysis

Principal Component Analysis (PCA) is a canonical and well-studied tool for dimension- ality reduction. However, when few data are available, the poor quality of the covariance estimator at its core may compromise its performance. We leverage this issue by casting the PCA into a multitask framework, and doing so, we show how to solve simultaneously several related PCA problems. Hence, we propose a novel formulation of the PCA prob- lem relying on a novel regularization. This regularization is based on a distance between subspaces, and the whole problem is solved as an optimization problem over a Riemannian manifold. We experimentally demonstrate the usefulness of our approach as pre-processing for EEG signals.

[1]  Bernhard Schölkopf,et al.  Kernel Principal Component Analysis , 1997, ICANN.

[2]  Daniel Boley,et al.  Common component analysis for multiple covariance matrices , 2011, KDD.

[3]  Antoine Souloumiac,et al.  Jacobi Angles for Simultaneous Diagonalization , 1996, SIAM J. Matrix Anal. Appl..

[4]  Karl Pearson F.R.S..  LIII. On lines and planes of closest fit to systems of points in space , 1901 .

[5]  Radford M. Neal.  Pattern Recognition and Machine Learning , 2007, Technometrics.

[6]  Robert D. Nowak,et al.  Online identification and tracking of subspaces from highly incomplete information , 2010, 2010 48th Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[7]  Xiaofei He,et al.  Locality Preserving Projections , 2003, NIPS.

[8]  Massimiliano Pontil,et al.  Convex multi-task feature learning , 2008, Machine Learning.

[9]  H. Hotelling.  Analysis of a complex of statistical variables into principal components. , 1933 .

[10]  Tijl De Bie,et al.  Eigenproblems in Pattern Recognition , 2005 .

[11]  Manfred K. Warmuth,et al.  Randomized PCA Algorithms with Regret Bounds that are Logarithmic in the Dimension , 2006, NIPS.

[12]  M. Grgic,et al.  A survey of biometric recognition methods , 2004, Proceedings. Elmar-2004. 46th International Symposium on Electronics in Marine.

[13]  Heng Tao Shen,et al.  Principal Component Analysis , 2009, Encyclopedia of Biometrics.

[14]  B. Moore.  Principal component analysis in linear systems: Controllability, observability, and model reduction , 1981 .

[15]  Christophe Pérignon,et al.  Why common factors in international bond returns are not so common , 2007 .

[16]  Massimiliano Pontil,et al.  Regularized multi--task learning , 2004, KDD.

[17]  Alan Edelman,et al.  The Geometry of Algorithms with Orthogonality Constraints , 1998, SIAM J. Matrix Anal. Appl..

[18]  Roland Badeau,et al.  Fast and Stable YAST Algorithm for Principal and Minor Subspace Tracking , 2008, IEEE Transactions on Signal Processing.

[19]  Bamdev Mishra,et al.  Manopt, a matlab toolbox for optimization on manifolds , 2013, J. Mach. Learn. Res..

[20]  Pascal Vincent,et al.  Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion , 2010, J. Mach. Learn. Res..

[21]  R. Fisher.  THE USE OF MULTIPLE MEASUREMENTS IN TAXONOMIC PROBLEMS , 1936 .

[22]  Masashi Sugiyama,et al.  Averaging covariance matrices for EEG signal classification based on the CSP: An empirical study , 2015, 2015 23rd European Signal Processing Conference (EUSIPCO).

[23]  David Zhang,et al.  Two-stage image denoising by principal component analysis with local pixel grouping , 2010, Pattern Recognit..

[24]  Dit-Yan Yeung,et al.  Multi-Task Learning in Heterogeneous Feature Spaces , 2011, AAAI.

[25]  S. J. Devlin,et al.  Robust Estimation of Dispersion Matrices and Principal Components , 1981 .

[26]  Florian Yger,et al.  A review of kernels on covariance matrices for BCI applications , 2013, 2013 IEEE International Workshop on Machine Learning for Signal Processing (MLSP).

[27]  Mehrtash Tafazzoli Harandi,et al.  From Manifold to Manifold: Geometry-Aware Dimensionality Reduction for SPD Matrices , 2014, ECCV.

[28]  G Pfurtscheller,et al.  Seperability of four-class motor imagery data using independent components analysis , 2006, Journal of neural engineering.

[29]  Yi Ma,et al.  Robust principal component analysis? , 2009, JACM.

[30]  Rich Caruana,et al.  Multitask Learning , 1997, Machine-mediated learning.

[31]  S. Shankar Sastry,et al.  Optimization Criteria and Geometric Algorithms for Motion and Structure Estimation , 2001, International Journal of Computer Vision.

[32]  Wojciech Samek,et al.  Transferring Subspaces Between Subjects in Brain--Computer Interfacing , 2012, IEEE Transactions on Biomedical Engineering.

[33]  Rabia Jafri,et al.  A Survey of Face Recognition Techniques , 2009, J. Inf. Process. Syst..

[34]  W. Gautschi,et al.  An algorithm for simultaneous orthogonal transformation of several positive definite symmetric matrices to nearly diagonal form , 1986 .

[35]  H. Hotelling.  Relations Between Two Sets of Variates , 1936 .

[36]  R. Tibshirani,et al.  Sparse Principal Component Analysis , 2006 .

[37]  Robert E. Mahony,et al.  Optimization Algorithms on Matrix Manifolds , 2007 .

[38]  Jean-Philippe Vert,et al.  Clustered Multi-Task Learning: A Convex Formulation , 2008, NIPS.

[39]  Jamal Atif,et al.  Metrics for Multivariate Dictionaries , 2013, ArXiv.

[40]  O. A. Fakolujo,et al.  A survey of face recognition techniques , 2007 .

[41]  Masashi Sugiyama,et al.  Geometry-aware principal component analysis for symmetric positive definite matrices , 2017, Machine Learning.

[42]  Moritz Grosse-Wentrup,et al.  Multisubject Learning for Common Spatial Patterns in Motor-Imagery BCI , 2011, Comput. Intell. Neurosci..

[43]  Rich Caruana,et al.  Multitask Learning , 1998, Encyclopedia of Machine Learning and Data Mining.

[44]  Joelle Pineau,et al.  Multitask Generalized Eigenvalue Program , 2016, AAAI.