Spectral Clustering Based on Local PCA

We propose a spectral clustering method based on local principal components analysis (PCA). After performing local PCA in selected neighborhoods, the algorithm builds a nearest neighbor graph weighted according to a discrepancy between the principal subspaces in the neighborhoods, and then applies spectral clustering. As opposed to standard spectral methods based solely on pairwise distances between points, our algorithm is able to resolve intersections. We establish theoretical guarantees for simpler variants within a prototypical mathematical framework for multi-manifold clustering, and evaluate our algorithm on various simulated data sets.

[1]  Emmanuel J. Candès,et al.  Robust Subspace Clustering , 2013, ArXiv.

[2]  Robert Pless,et al.  Manifold clustering , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[3]  Stephen Smale,et al.  Finding the Homology of Submanifolds with High Confidence from Random Samples , 2008, Discret. Comput. Geom..

[4]  Ulrike von Luxburg,et al.  A tutorial on spectral clustering , 2007, Stat. Comput..

[5]  W. Hoeffding.  Probability Inequalities for sums of Bounded Random Variables , 1963 .

[6]  René Vidal,et al.  A Unified Algebraic Approach to 2-D and 3-D Motion Segmentation and Estimation , 2006, Journal of Mathematical Imaging and Vision.

[7]  René Vidal,et al.  Filtrated Spectral Algebraic Subspace Clustering , 2015, 2015 IEEE International Conference on Computer Vision Workshop (ICCVW).

[8]  Wenbin Chen,et al.  Manifold clustering via energy minimization , 2007, Sixth International Conference on Machine Learning and Applications (ICMLA 2007).

[9]  Guangliang Chen,et al.  Foundations of a Multi-way Spectral Clustering Framework for Hybrid Linear Modeling , 2008, Found. Comput. Math..

[10]  Daniel N. Kaslovsky,et al.  Optimal Tangent Plane Recovery From Noisy Manifold Samples , 2011, ArXiv.

[11]  W. Kahan,et al.  The Rotation of Eigenvectors by a Perturbation. III , 1970 .

[12]  Pietro Perona,et al.  Self-Tuning Spectral Clustering , 2004, NIPS.

[13]  Ronen Basri,et al.  Lambertian Reflectance and Linear Subspaces , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[14]  A. Cuevas,et al.  On Statistical Properties of Sets Fulfilling Rolling-Type Conditions , 2011, Advances in Applied Probability.

[15]  René Vidal,et al.  Sparse Manifold Clustering and Embedding , 2011, NIPS.

[16]  Guangliang Chen,et al.  Kernel Spectral Curvature Clustering (KSCC) , 2009, 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops.

[17]  Serge J. Belongie,et al.  Higher order learning with graphs , 2006, ICML.

[18]  David J. Kriegman,et al.  Clustering appearances of objects under varying illumination conditions , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[19]  Michael I. Jordan,et al.  On Spectral Clustering: Analysis and an algorithm , 2001, NIPS.

[20]  René Vidal,et al.  Segmenting Motions of Different Types by Unsupervised Manifold Clustering , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[21]  Enn Saar,et al.  Statistics of the Galaxy Distribution , 2001 .

[22]  Allen Y. Yang,et al.  Estimation of Subspace Arrangements with Applications in Modeling and Segmenting Mixed Data , 2008, SIAM Rev..

[23]  Pietro Perona,et al.  Grouping and dimensionality reduction by locally linear embedding , 2001, NIPS.

[24]  Emmanuel J. Candès,et al.  A Geometric Analysis of Subspace Clustering with Outliers , 2011, ArXiv.

[25]  Guillermo Sapiro,et al.  Stratification Learning: Detecting Mixed Density and Dimensionality in High Dimensional Point Clouds , 2006, NIPS.

[26]  S. Afriat.  Orthogonal and oblique projectors and the characteristics of pairs of vector spaces , 1957, Mathematical Proceedings of the Cambridge Philosophical Society.

[27]  Achi Brandt,et al.  Fast multiscale clustering and manifold identification , 2006, Pattern Recognit..

[28]  Ulrike von Luxburg,et al.  Optimal construction of k-nearest-neighbor graphs for identifying noisy clusters , 2009, Theoretical Computer Science.

[29]  Russell A. Epstein,et al.  5/spl plusmn/2 eigenimages suffice: an empirical investigation of low-dimensional lighting models , 1995, Proceedings of the Workshop on Physics-Based Modeling in Computer Vision.

[30]  M. R. Brito,et al.  Connectivity of the mutual k-nearest-neighbor graph in clustering and outlier detection , 1997 .

[31]  Chandler Davis.  The rotation of eigenvectors by a perturbation , 1963 .

[32]  Michael I. Jordan,et al.  Mixtures of Probabilistic Principal Component Analyzers , 2001 .

[33]  Guangliang Chen,et al.  Spectral clustering based on local linear approximations , 2010, 1001.1323.

[34]  Guangliang Chen,et al.  Spectral Curvature Clustering (SCC) , 2009, International Journal of Computer Vision.

[35]  Jean-Daniel Boissonnat,et al.  Constructing Intrinsic Delaunay Triangulations of Submanifolds , 2013, ArXiv.

[36]  Aristides Gionis,et al.  Dimension induced clustering , 2005, KDD '05.

[37]  Robert D. Nowak,et al.  Multi-Manifold Semi-Supervised Learning , 2009, AISTATS.

[38]  Ery Arias-Castro,et al.  Clustering Based on Pairwise Distances When the Data is of Mixed Dimensions , 2009, IEEE Transactions on Information Theory.

[39]  Christopher M. Bishop,et al.  Mixtures of Probabilistic Principal Component Analyzers , 1999, Neural Computation.

[40]  Huan Xu,et al.  Noisy Sparse Subspace Clustering , 2013, J. Mach. Learn. Res..

[41]  G. Stewart,et al.  Matrix Perturbation Theory , 1990 .

[42]  Detection of non-random patterns in cosmological gravitational clustering , 2000, astro-ph/0011317.

[43]  Ayhan Demiriz,et al.  Constrained K-Means Clustering , 2000 .

[44]  Tieniu Tan,et al.  Similarity based vehicle trajectory clustering and anomaly detection , 2005, IEEE International Conference on Image Processing 2005.

[45]  Ying Wu,et al.  Multibody Grouping by Inference of Multiple Subspaces from High-Dimensional Data Using Oriented-Frames , 2006, IEEE Trans. Pattern Anal. Mach. Intell..

[46]  Helmut Bölcskei,et al.  Noisy subspace clustering via thresholding , 2013, 2013 IEEE International Symposium on Information Theory.

[47]  Gérard G. Medioni,et al.  Robust Multiple Manifolds Structure Learning , 2012, ICML 2012.

[48]  David J. Kriegman,et al.  Acquiring linear subspaces for face recognition under variable lighting , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[49]  Lawrence K. Saul,et al.  Think Globally, Fit Locally: Unsupervised Learning of Low Dimensional Manifold , 2003, J. Mach. Learn. Res..

[50]  Y. Jiang,et al.  Spectral Clustering on Multiple Manifolds , 2011, IEEE Transactions on Neural Networks.

[51]  M. Lachièze‐Rey,et al.  Statistics of the galaxy distribution , 1989 .

[52]  Pietro Perona,et al.  Beyond pairwise clustering , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[53]  Mauro Maggioni,et al.  Multiscale Estimation of Intrinsic Dimensionality of Data Sets , 2009, AAAI Fall Symposium: Manifold Learning and Its Applications.

[54]  Gilad Lerman,et al.  Hybrid Linear Modeling via Local Best-Fit Flats , 2010, International Journal of Computer Vision.

[55]  Joel A. Tropp,et al.  User-Friendly Tail Bounds for Sums of Random Matrices , 2010, Found. Comput. Math..

[56]  Tamir Hazan,et al.  Multi-way Clustering Using Super-Symmetric Non-negative Tensor Factorization , 2006, ECCV.