Dual Principal Component Pursuit

We consider the problem of outlier rejection in single subspace learning. Classical approaches work directly with a low-dimensional representation of the subspace. Our approach works with a dual representation of the subspace and hence aims to find its orthogonal complement. We pose this problem as an l1-minimization problem on the sphere and show that, under certain conditions on the distribution of the data, any global minimizer of this non-convex problem gives a vector orthogonal to the subspace. Moreover, we show that such a vector can still be found by relaxing the non-convex problem with a sequence of linear programs. Experiments on synthetic and real data show that the proposed approach, which we call Dual Principal Component Pursuit (DPCP), outperforms state-of-the art methods, especially in the case of high-dimensional subspaces.

[1]  E.J. Candes,et al.  An Introduction To Compressive Sampling , 2008, IEEE Signal Processing Magazine.

[2]  Yong Yu,et al.  Robust Subspace Segmentation by Low-Rank Representation , 2010, ICML.

[3]  Richard I. Hartley,et al.  In defence of the 8-point algorithm , 1995, Proceedings of IEEE International Conference on Computer Vision.

[4]  Robert F. Tichy,et al.  Spherical designs, discrepancy and numerical integration , 1993 .

[5]  John Wright,et al.  Finding a Sparse Vector in a Subspace: Linear Sparsity Using Alternating Directions , 2014, IEEE Transactions on Information Theory.

[6]  Daniel P. Robinson,et al.  Provable Self-Representation Based Outlier Detection in a Union of Subspaces , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Alberto Alzati,et al.  A Geometric Approach to the Trifocal Tensor , 2010, Journal of Mathematical Imaging and Vision.

[8]  Luke Oeding,et al.  The ideal of the trifocal variety , 2012, Math. Comput..

[9]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[10]  Yin Wang,et al.  Self Scaled Regularized Robust Regression , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Wotao Yin,et al.  Iteratively reweighted algorithms for compressive sensing , 2008, 2008 IEEE International Conference on Acoustics, Speech and Signal Processing.

[12]  Huan Wang,et al.  Exact Recovery of Sparsely-Used Dictionaries , 2012, COLT.

[13]  Shuicheng Yan,et al.  Online Robust PCA via Stochastic Optimization , 2013, NIPS.

[14]  René Vidal,et al.  Robust classification using structured sparse representation , 2011, CVPR 2011.

[15]  Heng Tao Shen,et al.  Principal Component Analysis , 2009, Encyclopedia of Biometrics.

[16]  Robert D. Nowak,et al.  Online identification and tracking of subspaces from highly incomplete information , 2010, 2010 48th Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[17]  John Wright,et al.  Complete Dictionary Recovery Over the Sphere I: Overview and the Geometric Picture , 2015, IEEE Transactions on Information Theory.

[18]  Joel A. Tropp,et al.  Robust Computation of Linear Models by Convex Relaxation , 2012, Foundations of Computational Mathematics.

[19]  Josef Dick.  Applications of geometric discrepancy in numerical analysis and statistics , 2014, Applied Algebra and Number Theory.

[20]  John Wright,et al.  Complete Dictionary Recovery Using Nonconvex Optimization , 2015, ICML.

[21]  R. Tichy,et al.  Discrepancies of Point Sequences on the Sphere and Numerical Integration , 2011 .

[22]  Pietro Perona,et al.  Learning Generative Visual Models from Few Training Examples: An Incremental Bayesian Approach Tested on 101 Object Categories , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[23]  José H. Dulá,et al.  A pure L1L1-norm principal component analysis , 2013, Comput. Stat. Data Anal..

[24]  Barak A. Pearlmutter,et al.  Blind Source Separation by Sparse Decomposition in a Signal Dictionary , 2001, Neural Computation.

[25]  Edmund Hlawka,et al.  Discrepancy and Riemann Integration , 1990 .

[26]  Karl Pearson F.R.S..  LIII. On lines and planes of closest fit to systems of points in space , 1901 .

[27]  John Wright,et al.  Complete dictionary recovery over the sphere , 2015, 2015 International Conference on Sampling Theory and Applications (SampTA).

[28]  W. Gander.  Least squares with a quadratic constraint , 1980 .

[29]  David J. Kriegman,et al.  From Few to Many: Illumination Cone Models for Face Recognition under Variable Lighting and Pose , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[30]  Michael Elad,et al.  The Cosparse Analysis Model and Algorithms , 2011, ArXiv.

[31]  René Vidal,et al.  Sparse Subspace Clustering: Algorithm, Theory, and Applications , 2012, IEEE transactions on pattern analysis and machine intelligence.

[32]  S. Lloyd,et al.  Quantum principal component analysis , 2013, Nature Physics.

[33]  B. Moore.  Principal component analysis in linear systems: Controllability, observability, and model reduction , 1981 .

[34]  I. Daubechies,et al.  Iteratively reweighted least squares minimization for sparse recovery , 2008, 0807.0575.

[35]  B. Mercier,et al.  A dual algorithm for the solution of nonlinear variational problems via finite element approximation , 1976 .

[36]  Gilad Lerman,et al.  $${l_p}$$lp-Recovery of the Most Significant Subspace Among Multiple Subspaces with Outliers , 2010, ArXiv.

[37]  Christos Georgakis,et al.  Disturbance detection and isolation by dynamic principal component analysis , 1995 .

[38]  René Vidal,et al.  Hyperplane Clustering via Dual Principal Component Pursuit , 2017, ICML.

[39]  Anmer Daskin.  Quantum Principal Component Analysis , 2015 .

[40]  Ronen Basri,et al.  Lambertian reflectance and linear subspaces , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[41]  David G. Lowe,et al.  Object recognition from local scale-invariant features , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[42]  Gilad Lerman,et al.  Fast, Robust and Non-convex Subspace Recovery , 2014, 1406.6145.

[43]  Constantine Caramanis,et al.  Robust PCA via Outlier Pursuit , 2010, IEEE Transactions on Information Theory.

[44]  Lilani Kumaranayake,et al.  Constructing socio-economic status indices: how to use principal components analysis. , 2006, Health policy and planning.

[45]  Glyn Harman,et al.  VARIATIONS ON THE KOKSMA-HLAWKA INEQUALITY , 2010 .

[46]  Johann S. Brauchart,et al.  Distributing many points on spheres: Minimal energy and designs , 2014, J. Complex..

[47]  G. Golub,et al.  Quadratically constrained least squares and quadratic problems , 1991 .

[48]  John Wright,et al.  Complete Dictionary Recovery Over the Sphere II: Recovery by Riemannian Trust-Region Method , 2015, IEEE Transactions on Information Theory.

[49]  Emmanuel J. Candès,et al.  A Geometric Analysis of Subspace Clustering with Outliers , 2011, ArXiv.

[50]  B. Ripley,et al.  Robust Statistics , 2018, Wiley Series in Probability and Statistics.

[51]  Gilad Lerman,et al.  A novel M-estimator for robust PCA , 2011, J. Mach. Learn. Res..

[52]  Robert C. Bolles,et al.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography , 1981, CACM.

[53]  Stephen P. Boyd,et al.  Enhancing Sparsity by Reweighted ℓ1 Minimization , 2007, 0711.1612.

[54]  Joe Kileel,et al.  Minimal Problems for the Calibrated Trifocal Variety , 2016, SIAM J. Appl. Algebra Geom..

[55]  J. Beck.  Sums of distances between points on a sphere — an application of the theory of irregularities of distribution to discrete Geometry , 1984 .

[56]  Lauwerens Kuipers,et al.  Uniform distribution of sequences , 1974 .

[57]  Arvind Ganesh,et al.  Fast Convex Optimization Algorithms for Exact Recovery of a Corrupted Low-Rank Matrix , 2009 .

[58]  Hans-Peter Kriegel,et al.  Subspace clustering , 2012, WIREs Data Mining Knowl. Discov..

[59]  G. Watson,et al.  On orthogonal linear ℓ1 approximation , 1987 .

[60]  George Atia,et al.  Coherence Pursuit: Fast, Simple, and Robust Principal Component Analysis , 2016, IEEE Transactions on Signal Processing.

[61]  Lars Eldèn.  Solving Quadratically Constrained Least Squares Problems Using a Differential-Geometric Approach , 2002 .

[62]  Michael Elad,et al.  From Sparse Solutions of Systems of Equations to Sparse Modeling of Signals and Images , 2009, SIAM Rev..

[63]  Yi Ma,et al.  Robust principal component analysis? , 2009, JACM.

[64]  H. Hotelling.  Analysis of a complex of statistical variables into principal components. , 1933 .

[65]  Chris H. Q. Ding,et al.  R1-PCA: rotational invariant L1-norm principal component analysis for robust subspace factorization , 2006, ICML.

[66]  Richard I. Hartley,et al.  In Defense of the Eight-Point Algorithm , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[67]  D. Reich,et al.  Principal components analysis corrects for stratification in genome-wide association studies , 2006, Nature Genetics.