EGGS: Eigen-Gap Guided Search Making Subspace Clustering Easy

The performance of spectral clustering heavily relies on the quality of affinity matrix. A variety of affinity-matrixconstruction methods have been proposed but they have hyperparameters to determine beforehand, which requires strong experience and lead to difficulty in real applications especially when the inter-cluster similarity is high or/and the dataset is large. On the other hand, we often have to determine to use a linear model or a nonlinear model, which still depends on experience. To solve these two problems, in this paper, we present an eigen-gap guided search method for subspace clustering. The main idea is to find the most reliable affinity matrix among a set of candidates constructed by linear and kernel regressions, where the reliability is quantified by the relative-eigen-gap of graph Laplacian defined in this paper. We show, theoretically and numerically, that the Laplacian matrix with a larger relative-eigen-gap often yields a higher clustering accuracy and stability. Our method is able to automatically search the best model and hyper-parameters in a pre-defined space. The search space is very easy to determine and can be arbitrarily large, though a relatively compact search space can reduce the highly unnecessary computation. Our method has high flexibility and convenience in real applications, and also has low computational cost because the affinity matrix is not computed by iterative optimization. We extend the method to large-scale datasets such as MNIST, on which the time cost is less than 90s and the clustering accuracy is state-of-the-art. Extensive experiments of natural image clustering show that our method is more stable, accurate, and efficient than baseline methods.

[1]  Xinlei Chen,et al.  Large Scale Spectral Clustering with Landmark-Based Representation , 2011, AAAI.

[2]  Xinlei Chen,et al.  Large Scale Spectral Clustering Via Landmark-Based Sparse Representation , 2015, IEEE Transactions on Cybernetics.

[3]  René Vidal,et al.  Kernel sparse subspace clustering , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[4]  Zhang Yi,et al.  Scalable Sparse Subspace Clustering , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[5]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[6]  Shin Matsushima,et al.  Selective Sampling-based Scalable Sparse Subspace Clustering , 2019, NeurIPS.

[7]  David J. Kriegman,et al.  Acquiring linear subspaces for face recognition under variable lighting , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[8]  Hongdong Li,et al.  Shape Interaction Matrix Revisited and Robustified: Efficient Subspace Clustering with Corrupted and Incomplete Data , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[9]  Zhihua Zhang,et al.  Exact Subspace Clustering in Linear Time , 2014, AAAI.

[10]  Zhang Yi,et al.  A Unified Framework for Representation-Based Subspace Clustering of Out-of-Sample and Large-Scale Data , 2013, IEEE Transactions on Neural Networks and Learning Systems.

[11]  René Vidal,et al.  Ieee Journal of Selected Topics in Signal Processing, Vol. X, No. X, Month 20xx 1 Latent Space Sparse and Low-rank Subspace Clustering , 2022 .

[12]  Jun Li,et al.  Large-scale Subspace Clustering by Fast Regression Coding , 2017, IJCAI.

[13]  Avinash C. Kak,et al.  PCA versus LDA , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[14]  Hongdong Li,et al.  Neural Collaborative Subspace Clustering , 2019, ICML.

[15]  Nathan Halko,et al.  Finding Structure with Randomness: Probabilistic Algorithms for Constructing Approximate Matrix Decompositions , 2009, SIAM Rev..

[16]  Ping Li,et al.  Learning Structured Low-Rank Representation via Matrix Factorization , 2016, AISTATS.

[17]  Andy Harter,et al.  Parameterisation of a stochastic model for human face identification , 1994, Proceedings of 1994 IEEE Workshop on Applications of Computer Vision.

[18]  Daniel P. Robinson,et al.  Scalable Sparse Subspace Clustering by Orthogonal Matching Pursuit , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  M. Fiedler Algebraic connectivity of graphs , 1973 .

[20]  Sameer A. Nene,et al.  Columbia Object Image Library (COIL100) , 1996 .

[21]  Chong You,et al.  Stochastic Sparse Subspace Clustering , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  René Vidal,et al.  Sparse Subspace Clustering: Algorithm, Theory, and Applications , 2012, IEEE transactions on pattern analysis and machine intelligence.

[23]  Daniel P. Robinson,et al.  Scalable Exemplar-based Subspace Clustering on Class-Imbalanced Data , 2018, European Conference on Computer Vision.

[24]  Shuicheng Yan,et al.  Robust and Efficient Subspace Segmentation via Least Squares Regression , 2012, ECCV.

[25]  Tong Zhang,et al.  Deep Subspace Clustering Networks , 2017, NIPS.

[26]  Wenbo Xu,et al.  Structured Graph Learning for Scalable Subspace Clustering: From Single View to Multiview , 2021, IEEE Transactions on Cybernetics.

[27]  Liang Xu,et al.  Regularized spectral learning , 2005, AISTATS.

[28]  Handong Zhao,et al.  Learnable Subspace Clustering , 2020, IEEE Transactions on Neural Networks and Learning Systems.

[29]  René Vidal,et al.  Structured Sparse Subspace Clustering: A unified optimization framework , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Stéphane Mallat,et al.  Invariant Scattering Convolution Networks , 2012, IEEE transactions on pattern analysis and machine intelligence.

[31]  Hans-Peter Kriegel,et al.  Subspace clustering , 2012, WIREs Data Mining Knowl. Discov..

[32]  Johannes Stallkamp,et al.  Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition , 2012, Neural Networks.

[33]  Roland Vollgraf,et al.  Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms , 2017, ArXiv.

[34]  René Vidal,et al.  A Structured Sparse Plus Structured Low-Rank Framework for Subspace Clustering and Completion , 2016, IEEE Transactions on Signal Processing.

[35]  Farhad Kamangar,et al.  Deep Low-Rank Subspace Clustering , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[36]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[37]  S. R. Searle,et al.  On Deriving the Inverse of a Sum of Matrices , 1981 .

[38]  Ulrike von Luxburg,et al.  A tutorial on spectral clustering , 2007, Stat. Comput..

[39]  Daniel P. Robinson,et al.  Oracle Based Active Set Algorithm for Scalable Elastic Net Subspace Clustering , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[40]  Jian Pei,et al.  Finding multiple stable clusterings , 2016, Knowledge and Information Systems.

[41]  Yong Yu,et al.  Robust Recovery of Subspace Structures by Low-Rank Representation , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[42]  Zhouchen Lin,et al.  Self-Supervised Convolutional Subspace Clustering Network , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[43]  Tao Mei,et al.  Subspace Clustering by Block Diagonal Representation , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[44]  Hongdong Li,et al.  Efficient dense subspace clustering , 2014, IEEE Winter Conference on Applications of Computer Vision.