Unsupervised deep learning for subspace clustering

This paper presents a novel technique for the segmentation of data W = [w<inf>1</inf> · · · w<inf>n</inf>] ⊂ R<sup>D</sup> drawn from a union u = ∪<sup>M</sup><inf>i=1</inf> of subspaces {S<inf>i</inf>}<sup>M</sup><inf>i=1</inf>. First, an existing subspace segmentation algorithm is used to perform an initial data clustering {C<inf>i</inf>}<sup>M</sup><inf>i=1</inf>, where C<inf>i</inf> = {w<inf>i1</inf> · · ·w<inf>ik</inf>} ⊂ W is the set of data from the i<sup>th</sup> cluster. Then, a local subspace LS<inf>i</inf> is matched for each C<inf>i</inf> and the distance d<inf>ij</inf> between LS<inf>i</inf> and each point w<inf>ij</inf> ∊ C<inf>i</inf> is computed. A data-driven threshold η is computed and the data points (in C<inf>i</inf>) whose distances to LS<inf>i</inf> are larger than η are eliminated since they are considered as outliers or erroneously clustered data points in C<inf>i</inf>. The remaining data points C<inf>i</inf> ⊂ C<inf>i</inf> are considered to be coming from the same subspace with high confidence. Then, {C<inf>i</inf>}<sup>M</sup><inf>i=1</inf> are used in unsupervised way to train a convolution neural network to obtain a deep learning model, which is in turn used to re-cluster W. The system has been successfully implemented using the MNIST dataset and it improved the segmentation accuracy of a particular algorithm (EnSC-ORGEN) from 93.79% to 96.52%.

[1]  Guangliang Chen,et al.  Spectral Curvature Clustering (SCC) , 2009, International Journal of Computer Vision.

[2]  P. Tseng.  Nearest q-Flat to m Points , 2000 .

[3]  D. Hubel,et al.  Receptive fields of single neurones in the cat's striate cortex , 1959, The Journal of physiology.

[4]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Yonina C. Eldar,et al.  Robust Recovery of Signals From a Structured Union of Subspaces , 2008, IEEE Transactions on Information Theory.

[6]  Marc Pollefeys,et al.  A General Framework for Motion Segmentation: Independent, Articulated, Rigid, Non-rigid, Degenerate and Non-degenerate , 2006, ECCV.

[7]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[8]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[9]  R. Vidal.  A TUTORIAL ON SUBSPACE CLUSTERING , 2010 .

[10]  S. Mallat,et al.  Invariant Scattering Convolution Networks , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[11]  René Vidal,et al.  Filtrated Spectral Algebraic Subspace Clustering , 2015, 2015 IEEE International Conference on Computer Vision Workshop (ICCVW).

[12]  Y. Wong.  Differential geometry of grassmann manifolds. , 1967, Proceedings of the National Academy of Sciences of the United States of America.

[13]  René Vidal,et al.  Sparse subspace clustering , 2009, CVPR.

[14]  João Paulo Costeira,et al.  Subspace segmentation with outliers: A grassmannian approach to the maximum consensus subspace , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[15]  S. Shankar Sastry,et al.  Generalized principal component analysis (GPCA) , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[16]  Andreas Mayr,et al.  Deep Learning as an Opportunity in Virtual Screening , 2015 .

[17]  Sebastian Thrun,et al.  Dermatologist-level classification of skin cancer with deep neural networks , 2017, Nature.

[18]  K. Kanatani,et al.  Estimating the Number of Independent Motions for Multibody Motion Segmentation , 2002 .

[19]  Yong Yu,et al.  Robust Subspace Segmentation by Low-Rank Representation , 2010, ICML.

[20]  Ronen Basri,et al.  Lambertian Reflectance and Linear Subspaces , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[21]  Akram Aldroubi,et al.  Nonlinear Least Squares in ℝN , 2009 .

[22]  David J. Kriegman,et al.  Clustering appearances of objects under varying illumination conditions , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[23]  Gene H. Golub,et al.  Matrix computations , 1983 .

[24]  Joaquim Salvi,et al.  Enhanced Local Subspace Affinity for feature-based motion segmentation , 2011, Pattern Recognit..

[25]  Akram Aldroubi,et al.  Optimal Non-Linear Models for Sparsity and Sampling , 2007, 0707.2008.

[26]  René Vidal,et al.  Clustering disjoint subspaces via sparse representation , 2010, 2010 IEEE International Conference on Acoustics, Speech and Signal Processing.

[27]  Rob Fergus,et al.  Visualizing and Understanding Convolutional Networks , 2013, ECCV.

[28]  Daniel P. Robinson,et al.  Scalable Sparse Subspace Clustering by Orthogonal Matching Pursuit , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  Allen Y. Yang,et al.  Estimation of Subspace Arrangements with Applications in Modeling and Segmenting Mixed Data , 2008, SIAM Rev..