Determining the Number of Clusters via Iterative Consensus Clustering

We use a cluster ensemble to determine the number of clusters, k, in a group of data. A consensus similarity matrix is formed from the ensemble using multiple algorithms and several values for k. A random walk is induced on the graph defined by the consensus matrix and the eigenvalues of the associated transition probability matrix are used to determine the number of clusters. For noisy or high-dimensional data, an iterative technique is presented to refine this consensus matrix in way that encourages a block-diagonal form. It is shown that the resulting consensus matrix is generally superior to existing similarity matrices for this type of spectral analysis.

[1]  Ulrike von Luxburg,et al.  A tutorial on spectral clustering , 2007, Stat. Comput..

[2]  William W. Cohen,et al.  Power Iteration Clustering , 2010, ICML.

[3]  Daniel Boley,et al.  Principal Direction Divisive Partitioning , 1998, Data Mining and Knowledge Discovery.

[4]  Miguel Á. Carreira-Perpiñán,et al.  Fast nonparametric clustering with Gaussian blurring mean-shift , 2006, ICML.

[5]  Amy Nicole Langville,et al.  Algorithms, Initializations, and Convergence for the Nonnegative Matrix Factorization , 2014, ArXiv.

[6]  Carl Dean Meyer,et al.  Stochastic Data Clustering , 2010, SIAM J. Matrix Anal. Appl..

[7]  Gerard Salton,et al.  Term-Weighting Approaches in Automatic Text Retrieval , 1988, Inf. Process. Manag..

[8]  Carl D. Meyer,et al.  Matrix Analysis and Applied Linear Algebra , 2000 .

[9]  Jill P. Mesirov,et al.  Consensus Clustering: A Resampling-Based Method for Class Discovery and Visualization of Gene Expression Microarray Data , 2003, Machine Learning.

[10]  P. Deuflhard,et al.  Identification of almost invariant aggregates in reversible nearly uncoupled Markov chains , 2000 .

[11]  Michael I. Jordan,et al.  On Spectral Clustering: Analysis and an algorithm , 2001, NIPS.

[12]  Jianbo Shi,et al.  A Random Walks View of Spectral Segmentation , 2001, AISTATS.

[13]  R. Larsen.  Lanczos Bidiagonalization With Partial Reorthogonalization , 1998 .

[14]  Daniel T. Larose,et al.  Discovering Knowledge in Data: An Introduction to Data Mining , 2005 .

[15]  SaltonGerard,et al.  Term-weighting approaches in automatic text retrieval , 1988 .

[16]  Jitendra Malik,et al.  Normalized cuts and image segmentation , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[17]  J. Kogan.  Introduction to Clustering Large and High-Dimensional Data , 2007 .

[18]  Lada A. Adamic,et al.  The political blogosphere and the 2004 U.S. election: divided they blog , 2005, LinkKDD '05.

[19]  Jiawei Han,et al.  Spectral Clustering , 2018, Data Clustering: Algorithms and Applications.

[20]  André Carlos Ponce de Leon Ferreira de Carvalho,et al.  Consensus Clustering Using Spectral Theory , 2008, ICONIP.

[21]  Michael W. Berry,et al.  Algorithms and applications for approximate nonnegative matrix factorization , 2007, Comput. Stat. Data Anal..

[22]  William J. Stewart,et al.  Probability, Markov Chains, Queues, and Simulation: The Mathematical Basis of Performance Modeling , 2009 .

[23]  H. Sebastian Seung,et al.  Learning the parts of objects by non-negative matrix factorization , 1999, Nature.

[24]  B. Nordstrom.  FINITE MARKOV CHAINS , 2005 .

[25]  John G. Kemeny,et al.  Finite Markov chains , 1960 .