Sparse kernel models for spectral clustering using the incomplete Cholesky decomposition

A new sparse kernel model for spectral clustering is presented. This method is based on the incomplete Cholesky decomposition and can be used to efficiently solve large-scale spectral clustering problems. The formulation arises from a weighted kernel principal component analysis (PCA) interpretation of spectral clustering. The interpretation is within a constrained optimization framework with primal and dual model representations allowing the clustering model to be extended to out-of-sample points. The incomplete Cholesky decomposition is used to compute low-rank approximations of a modified affinity matrix derived from the data which contains cluster information. A reduced set method is also presented to compute efficiently the cluster indicators for out-of-sample data. Simulation results with large-scale toy datasets and images show improved performance in terms of computational complexity.

[1]  Pietro Perona,et al.  Self-Tuning Spectral Clustering , 2004, NIPS.

[2]  Nicolas Le Roux,et al.  Out-of-Sample Extensions for LLE, Isomap, MDS, Eigenmaps, and Spectral Clustering , 2003, NIPS.

[3]  Stephen J. Wright.  Modified Cholesky Factorizations in Interior-Point Algorithms for Linear Programming , 1999, SIAM J. Optim..

[4]  John Hallam,et al.  IEEE International Joint Conference on Neural Networks , 2005 .

[5]  Katya Scheinberg,et al.  Efficient SVM Training Using Low-Rank Kernel Representations , 2002, J. Mach. Learn. Res..

[6]  Chris H. Q. Ding,et al.  K-means clustering via principal component analysis , 2004, ICML.

[7]  Gene H. Golub,et al.  Matrix computations , 1983 .

[8]  Michael I. Jordan,et al.  On Spectral Clustering: Analysis and an algorithm , 2001, NIPS.

[9]  K. Fan.  On a Theorem of Weyl Concerning Eigenvalues of Linear Transformations I. , 1949, Proceedings of the National Academy of Sciences of the United States of America.

[10]  K. Fan.  On a Theorem of Weyl Concerning Eigenvalues of Linear Transformations: II. , 1949, Proceedings of the National Academy of Sciences of the United States of America.

[11]  Michael I. Jordan,et al.  Kernel independent component analysis , 2003, 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings. (ICASSP '03)..

[12]  Johan A. K. Suykens,et al.  Multiway Spectral Clustering with Out-of-Sample Extensions through Weighted Kernel PCA , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  Johan A. K. Suykens,et al.  A support vector machine formulation to PCA analysis and its kernel version , 2003, IEEE Trans. Neural Networks.

[14]  Johan A. K. Suykens,et al.  Joint Conference on Neural Networks , Orlando , Florida , USA , August 12-17 , 2007 ICA through an LS-SVM based Kernel CCA Measure for Independence , 2007 .

[15]  R. Shah,et al.  Least Squares Support Vector Machines , 2022 .

[16]  Jianbo Shi,et al.  A Random Walks View of Spectral Segmentation , 2001, AISTATS.

[17]  Michael I. Jordan,et al.  Learning Spectral Clustering, With Application To Speech Separation , 2006, J. Mach. Learn. Res..

[18]  Martine D. F. Schlag,et al.  Spectral K-way ratio-cut partitioning and clustering , 1994, IEEE Trans. Comput. Aided Des. Integr. Circuits Syst..

[19]  Christopher J. C. Burges,et al.  Simplified Support Vector Decision Rules , 1996, ICML.

[20]  Thomas G. Dietterich,et al.  Solving Multiclass Learning Problems via Error-Correcting Output Codes , 1994, J. Artif. Intell. Res..

[21]  Jitendra Malik,et al.  A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[22]  Johan A. K. Suykens,et al.  A Weighted Kernel PCA Formulation with Out-of-Sample Extensions for Spectral Clustering Methods , 2006, The 2006 IEEE International Joint Conference on Neural Network Proceedings.

[23]  Jitendra Malik,et al.  Normalized cuts and image segmentation , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[24]  Chris H. Q. Ding,et al.  Linearized cluster assignment via spectral ordering , 2004, ICML.

[25]  M. Fiedler.  A property of eigenvectors of nonnegative symmetric matrices and its application to graph theory , 1975 .

[26]  Johan A. K. Suykens,et al.  A regularized kernel CCA contrast function for ICA , 2008, Neural Networks.

[27]  U. Feige,et al.  Spectral Graph Theory , 2015 .

[28]  Michael I. Jordan,et al.  Learning Spectral Clustering , 2003, NIPS.

[29]  Charles R. Johnson,et al.  Matrix analysis , 1985, Statistical Inference for Engineers and Data Scientists.

[30]  Bernhard Schölkopf,et al.  Kernel Methods for Measuring Independence , 2005, J. Mach. Learn. Res..

[31]  Gunnar Rätsch,et al.  Input space versus feature space in kernel-based methods , 1999, IEEE Trans. Neural Networks.

[32]  C. Ding,et al.  Spectral relaxation models and structure analysis for K-way graph clustering and bi-clustering , 2001 .

[33]  Andrew B. Kahng,et al.  New spectral methods for ratio cut partitioning and clustering , 1991, IEEE Trans. Comput. Aided Des. Integr. Circuits Syst..