Graph nodes clustering with the sigmoid commute-time kernel: A comparative study

This work addresses the problem of detecting clusters in a weighted, undirected, graph by using kernel-based clustering methods, directly partitioning the graph according to a well-defined similarity measure between the nodes (a kernel on a graph). The proposed algorithms are based on a two-step procedure. First, a kernel or similarity matrix, providing a meaningful similarity measure between any couple of nodes, is computed from the adjacency matrix of the graph. Then, the nodes of the graph are clustered by performing a kernel clustering on this similarity matrix. Besides the introduction of a prototype-based kernel version of the gaussian mixtures model and Ward's hierarchical clustering, in addition to the already known kernel k-means and fuzzy k-means, a new kernel, called the sigmoid commute-time kernel (K"C"T^S) is presented. The joint use of the K"C"T^S kernel matrix and kernel clustering appears to be quite effective. Indeed, this methodology provides the best results on a systematic comparison with a selection of graph clustering and communities detection algorithms on three real-world databases. Finally, some links between the proposed hierarchical kernel clustering and spectral clustering are examined.

[1]  Francesco Masulli,et al.  A survey of kernel and spectral methods for clustering , 2008, Pattern Recognit..

[2]  Alan M. Frieze,et al.  Clustering Large Graphs via the Singular Value Decomposition , 2004, Machine Learning.

[3]  Andrew McCallum,et al.  Automating the Construction of Internet Portals with Machine Learning , 2000, Information Retrieval.

[4]  Jitendra Malik,et al.  Normalized Cuts and Image Segmentation , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[5]  Martin Rosvall,et al.  An information-theoretic framework for resolving community structure in complex networks , 2007, Proceedings of the National Academy of Sciences.

[6]  François Fouss,et al.  Graph Nodes Clustering Based on the Commute-Time Kernel , 2007, PAKDD.

[7]  Pavel Yu. Chebotarev,et al.  The Forest Metrics for Graph Vertices , 2002, Electron. Notes Discret. Math..

[8]  Pierre Dupont,et al.  Learning Partially Observable Markov Models from First Passage Times , 2007, ECML.

[9]  Chris H. Q. Ding,et al.  Spectral Relaxation for K-means Clustering , 2001, NIPS.

[10]  G. Caldarelli,et al.  Detecting communities in large networks , 2004, cond-mat/0402499.

[11]  Anton J. Enright,et al.  An efficient algorithm for large-scale detection of protein families. , 2002, Nucleic acids research.

[12]  A. Arenas,et al.  Community detection in complex networks using extremal optimization. , 2005, Physical review. E, Statistical, nonlinear, and soft matter physics.

[13]  Katya Scheinberg,et al.  Efficient SVM Training Using Low-Rank Kernel Representations , 2002, J. Mach. Learn. Res..

[14]  Paul Van Dooren,et al.  A MEASURE OF SIMILARITY BETWEEN GRAPH VERTICES . WITH APPLICATIONS TO SYNONYM EXTRACTION AND WEB SEARCHING , 2002 .

[15]  Massimo Marchiori,et al.  Method to find community structures based on information centrality. , 2004, Physical review. E, Statistical, nonlinear, and soft matter physics.

[16]  Xiang-Sun Zhang,et al.  Graph kernels, hierarchical clustering, and network community structure: experiments and comparative analysis , 2007 .

[17]  Francesco Camastra,et al.  A Novel Kernel Method for Clustering , 2005, IEEE Trans. Pattern Anal. Mach. Intell..

[18]  Matthieu Latapy,et al.  Computing Communities in Large Networks Using Random Walks , 2004, J. Graph Algorithms Appl..

[19]  Nello Cristianini,et al.  Learning Semantic Similarity , 2002, NIPS.

[20]  P. Tetali Random walks and the effective resistance of networks , 1991 .

[21]  Mark Newman,et al.  Detecting community structure in networks , 2004 .

[22]  Jianbo Shi,et al.  A Random Walks View of Spectral Segmentation , 2001, AISTATS.

[23]  Haluk Bingol,et al.  Community Detection in Complex Networks Using Genetic Algorithms , 2006, 0711.0491.

[24]  Colin Fyfe,et al.  The kernel self-organising map , 2000, KES'2000. Fourth International Conference on Knowledge-Based Intelligent Engineering Systems and Allied Technologies. Proceedings (Cat. No.00TH8516).

[25]  Yuji Matsumoto,et al.  Application of Kernels to Link Analysis: First Results , 2004 .

[26]  Haijun Zhou Distance, dissimilarity index, and network community structure. , 2003, Physical review. E, Statistical, nonlinear, and soft matter physics.

[27]  Chris H. Q. Ding,et al.  Nonnegative Lagrangian Relaxation of K-Means and Spectral Clustering , 2005, ECML.

[28]  F. Chung Laplacians and the Cheeger Inequality for Directed Graphs , 2005 .

[29]  Charu C. Aggarwal,et al.  Graph Clustering , 2010, Encyclopedia of Machine Learning and Data Mining.

[30]  Bojan Mohar,et al.  Laplace eigenvalues of graphs - a survey , 1992, Discret. Math..

[31]  H. H. Gan,et al.  RAG: RNA-As-Graphs database-concepts, analysis, features , 2004, Bioinform..

[32]  Padhraic Smyth,et al.  A Spectral Clustering Approach To Finding Communities in Graph , 2005, SDM.

[33]  Padhraic Smyth,et al.  Algorithms for estimating relative importance in networks , 2003, KDD '03.

[34]  Yannis Manolopoulos,et al.  Indexed-based density biased sampling for clustering applications , 2006, Data Knowl. Eng..

[35]  David Liben-Nowell,et al.  The link-prediction problem for social networks , 2007 .

[36]  Henry G. Small,et al.  Co-citation in the scientific literature: A new measure of the relationship between two documents , 1973, J. Am. Soc. Inf. Sci..

[37]  Edwin R. Hancock,et al.  Spectral embedding of graphs , 2003, Pattern Recognit..

[38]  Stefan Bornholdt,et al.  Detecting fuzzy community structures in complex networks with a Potts model. , 2004, Physical review letters.

[39]  Jacek M. Leski,et al.  Towards a robust fuzzy clustering , 2003, Fuzzy Sets Syst..

[40]  Purnamrita Sarkar,et al.  A Tractable Approach to Finding Closest Truncated-commute-time Neighbors in Large Graphs , 2007, UAI.

[41]  Kun Huang,et al.  A unifying theorem for spectral embedding and clustering , 2003, AISTATS.

[42]  Santosh S. Vempala,et al.  On clusterings: Good, bad and spectral , 2004, JACM.

[43]  Stéphane Lafon,et al.  Diffusion maps , 2006 .

[44]  Andrew Zisserman,et al.  Advances in Neural Information Processing Systems (NIPS) , 2007 .

[45]  M. A. Muñoz,et al.  Journal of Statistical Mechanics: An IOP and SISSA journal Theory and Experiment Detecting network communities: a new systematic and efficient algorithm , 2004 .

[46]  M E J Newman,et al.  Fast algorithm for detecting community structure in networks. , 2003, Physical review. E, Statistical, nonlinear, and soft matter physics.

[47]  Risi Kondor,et al.  Diffusion kernels on graphs and other discrete structures , 2002, ICML 2002.

[48]  Leon Danon,et al.  Comparing community structure identification , 2005, cond-mat/0505245.

[49]  Satu Elisa Schaeffer,et al.  Graph Clustering , 2017, Encyclopedia of Machine Learning and Data Mining.

[50]  Miin-Shen Yang,et al.  Alternative c-means clustering algorithms , 2002, Pattern Recognit..

[51]  Pavel Yu. Chebotarev,et al.  Spanning Forests of a Digraph and Their Applications , 2001, ArXiv.

[52]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[53]  Thomas Gärtner,et al.  Cyclic pattern kernels for predictive graph mining , 2004, KDD.

[54]  John D. Lafferty,et al.  Diffusion Kernels on Graphs and Other Discrete Input Spaces , 2002, ICML.

[55]  Kenneth G. Manton,et al.  Fuzzy Cluster Analysis , 2005 .

[56]  Lawrence B. Holder,et al.  Mining Graph Data , 2006 .

[57]  N. Alves Unveiling community structures in weighted networks. , 2007, Physical review. E, Statistical, nonlinear, and soft matter physics.

[58]  S.,et al.  An Efficient Heuristic Procedure for Partitioning Graphs , 2022 .

[59]  Leo Grady,et al.  Random Walks for Image Segmentation , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[60]  Christos Faloutsos,et al.  Electricity Based External Similarity of Categorical Attributes , 2003, PAKDD.

[61]  Deli Zhao,et al.  Contextual Distance for Data Perception , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[62]  Hinrich Schütze,et al.  Introduction to information retrieval , 2008 .

[63]  Christopher J. Fox,et al.  Lexical Analysis and Stoplists , 1992, Information Retrieval: Data Structures & Algorithms.

[64]  Sergios Theodoridis,et al.  Pattern Recognition, Third Edition , 2006 .

[65]  Béla Bollobás,et al.  Modern Graph Theory , 2002, Graduate Texts in Mathematics.

[66]  S. V. N. Vishwanathan,et al.  Graph kernels , 2007 .

[67]  Mark A. Girolami,et al.  Mercer kernel-based clustering in feature space , 2002, IEEE Trans. Neural Networks.

[68]  François Fouss,et al.  An Experimental Investigation of Graph Kernels on a Collaborative Recommendation Task , 2006, Sixth International Conference on Data Mining (ICDM'06).

[69]  Jonathan D. Cohen,et al.  Drawing graphs to convey proximity: an incremental arrangement method , 1997, TCHI.

[70]  R. Guimerà,et al.  Functional cartography of complex metabolic networks , 2005, Nature.

[71]  Jennifer Neville,et al.  Collective Classification with Relational Dependency Networks , 2003 .

[72]  Alexander J. Smola,et al.  Kernels and Regularization on Graphs , 2003, COLT.

[73]  Péter András Kernel-Kohonen Networks , 2002, Int. J. Neural Syst..

[74]  M. M. Kessler Bibliographic coupling between scientific papers , 1963 .

[75]  Matthew Brand,et al.  A Random Walks Perspective on Maximizing Satisfaction and Profit , 2005, SDM.

[76]  Inderjit S. Dhillon,et al.  Concept Decompositions for Large Sparse Text Data Using Clustering , 2004, Machine Learning.

[77]  Kohei Inoue,et al.  Robust Kernel Fuzzy Clustering , 2005, FSKD.

[78]  P. Sopp Cluster analysis. , 1996, Veterinary immunology and immunopathology.

[79]  Paul Van Dooren,et al.  A measure of similarity between graph vertices , 2004 .

[80]  Christos Faloutsos,et al.  Random walk with restart: fast solutions and applications , 2008, Knowledge and Information Systems.

[81]  Zhongdong Wu,et al.  Fuzzy C-means clustering algorithm based on kernel method , 2003, Proceedings Fifth International Conference on Computational Intelligence and Multimedia Applications. ICCIMA 2003.

[82]  Sergios Theodoridis,et al.  Pattern Recognition , 1998, IEEE Trans. Neural Networks.

[83]  Ulrike von Luxburg,et al.  A tutorial on spectral clustering , 2007, Stat. Comput..

[84]  M. Newman,et al.  Finding community structure in networks using the eigenvectors of matrices. , 2006, Physical review. E, Statistical, nonlinear, and soft matter physics.

[85]  Sadaaki Miyamoto,et al.  LVQ Clustering and SOM Using a Kernel Function , 2005 .

[86]  Prabhakar Raghavan,et al.  The electrical resistance of a graph captures its commute and cover times , 2005, computational complexity.

[87]  Thomas Sauerwald,et al.  A new diffusion-based multilevel algorithm for computing graph partitions of very high quality , 2008, 2008 IEEE International Symposium on Parallel and Distributed Processing.

[88]  Evangelos E. Milios,et al.  Node similarity in the citation graph , 2006, Knowledge and Information Systems.

[89]  Fang Wu,et al.  Finding communities in linear time: a physics approach , 2003, ArXiv.

[90]  François Fouss,et al.  Random-Walk Computation of Similarities between Nodes of a Graph with Application to Collaborative Recommendation , 2007, IEEE Transactions on Knowledge and Data Engineering.

[91]  Dao-Qiang Zhang,et al.  A novel kernelized fuzzy C-means algorithm with application in medical image segmentation , 2004, Artif. Intell. Medicine.

[92]  Rajeev Motwani,et al.  The PageRank Citation Ranking : Bringing Order to the Web , 1999, WWW 1999.

[93]  M E J Newman,et al.  Community structure in social and biological networks , 2001, Proceedings of the National Academy of Sciences of the United States of America.

[94]  Sergey Brin,et al.  The Anatomy of a Large-Scale Hypertextual Web Search Engine , 1998, Comput. Networks.

[95]  Martin F. Porter,et al.  An algorithm for suffix stripping , 1997, Program.

[96]  David Harel,et al.  On Clustering Using Random Walks , 2001, FSTTCS.

[97]  Alejandro Murua,et al.  Hierarchical model-based clustering of large datasets through fractionation and refractionation , 2004, Inf. Syst..

[98]  Patrick Brézillon,et al.  Lecture Notes in Artificial Intelligence , 1999 .

[99]  Zoubin Ghahramani,et al.  Graph Kernels by Spectral Transforms , 2006, Semi-Supervised Learning.

[100]  Pavel Yu. Chebotarev,et al.  The Forest Metric for Graph Vertices , 2006, ArXiv.

[101]  Ann B. Lee,et al.  Diffusion maps and coarse-graining: a unified framework for dimensionality reduction, graph partitioning, and data set parameterization , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[102]  J. C. Peters,et al.  Fuzzy Cluster Analysis : A New Method to Predict Future Cardiac Events in Patients With Positive Stress Tests , 1998 .

[103]  XuLei Yang,et al.  A robust deterministic annealing algorithm for data clustering , 2005, Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005..

[104]  J. N. Kapur,et al.  Entropy optimization principles with applications , 1992 .

[105]  Alexander J. Smola,et al.  Learning with kernels , 1998 .

[106]  Chris H. Q. Ding,et al.  Linearized cluster assignment via spectral ordering , 2004, ICML.

[107]  Yuji Matsumoto,et al.  Application of kernels to link analysis , 2005, KDD '05.

[108]  Luh Yen,et al.  A family of dissimilarity measures between nodes generalizing both the shortest-path and the commute-time distances , 2008, KDD.

[109]  Ricardo Baeza-Yates,et al.  Information Retrieval: Data Structures and Algorithms , 1992 .

[110]  Christos Faloutsos,et al.  Automatic multimedia cross-modal correlation discovery , 2004, KDD.

[111]  M. Randic,et al.  Resistance distance , 1993 .

[112]  François Fouss,et al.  The Principal Components Analysis of a Graph, and Its Relationships to Spectral Clustering , 2004, ECML.

[113]  Mo Chen,et al.  Directed Graph Embedding , 2007, IJCAI.

[114]  J. H. Ward Hierarchical Grouping to Optimize an Objective Function , 1963 .

[115]  Joachim M. Buhmann,et al.  Going Metric: Denoising Pairwise Data , 2002, NIPS.

[116]  S. Borgatti,et al.  LS sets, lambda sets and other cohesive subsets , 1990 .

[117]  Hidetomo Ichihashi,et al.  Gaussian Mixture PDF Approximation and Fuzzy c-Means Clustering with Entropy Regularization , 2000 .

[118]  T. Landauer,et al.  Indexing by Latent Semantic Analysis , 1990 .

[119]  M. Fiedler A property of eigenvectors of nonnegative symmetric matrices and its application to graph theory , 1975 .

[120]  Anil K. Jain,et al.  Algorithms for Clustering Data , 1988 .

[121]  Chris H. Q. Ding,et al.  A learning framework using Green's function and kernel regularization with application to recommender system , 2007, KDD '07.

[122]  Daoqiang Zhang,et al.  Learning the kernel parameters in kernel minimum distance classifier , 2006, Pattern Recognit..

[123]  Nello Cristianini,et al.  Kernel Methods for Pattern Analysis , 2004 .

[124]  Yair Weiss,et al.  Segmentation using eigenvectors: a unifying view , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[125]  Bernhard Schölkopf,et al.  A kernel view of the dimensionality reduction of manifolds , 2004, ICML.

[126]  Robert E. Tarjan,et al.  Graph Clustering and Minimum Cut Trees , 2004, Internet Math..

[127]  Bernhard Schölkopf,et al.  Learning from labeled and unlabeled data on a directed graph , 2005, ICML.

[128]  Tony F. Chan,et al.  On the Optimality of the Median Cut Spectral Bisection Graph Partitioning Method , 1997, SIAM J. Sci. Comput..

[129]  Doheon Lee,et al.  Evaluation of the performance of clustering algorithms in kernel-induced feature space , 2005, Pattern Recognit..

[130]  Leo Grady,et al.  Isoperimetric Partitioning: A New Algorithm for Graph Partitioning , 2005, SIAM J. Sci. Comput..

[131]  Chris H. Q. Ding,et al.  A min-max cut algorithm for graph partitioning and data clustering , 2001, Proceedings 2001 IEEE International Conference on Data Mining.

[132]  Marco Gori,et al.  A random-walk based scoring algorithm with application to recommender systems for large-scale e-commerce , 2006, KDD 2006.

[133]  Edwin R. Hancock,et al.  Clustering and Embedding Using Commute Times , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[134]  A. Seary,et al.  Partitioning Networks by Eigenvectors , 2004 .

[135]  Donald Gustafson,et al.  Fuzzy clustering with a fuzzy covariance matrix , 1978, 1978 IEEE Conference on Decision and Control including the 17th Symposium on Adaptive Processes.

[136]  T. Schlick,et al.  RAG: RNA-As-Graphs database—concepts, analysis, and features , 1987 .

[137]  Alex Pothen,et al.  PARTITIONING SPARSE MATRICES WITH EIGENVECTORS OF GRAPHS* , 1990 .

[138]  Daoqiang Zhang,et al.  Fuzzy clustering using kernel method , 2002 .

[139]  S. vanDongen Graph Clustering by Flow Simulation , 2000 .

[140]  Hava T. Siegelmann,et al.  Support Vector Clustering , 2002, J. Mach. Learn. Res..

[141]  I. Dhillon,et al.  A Unified View of Kernel k-means , Spectral Clustering and Graph Cuts , 2004 .

[142]  Andrew B. Kahng,et al.  New spectral methods for ratio cut partitioning and clustering , 1991, IEEE Trans. Comput. Aided Des. Integr. Circuits Syst..

[143]  Martine D. F. Schlag,et al.  Multi-level spectral hypergraph partitioning with arbitrary vertex sizes , 1996, Proceedings of International Conference on Computer Aided Design.

[144]  Yehuda Koren,et al.  Measuring and extracting proximity in networks , 2006, KDD '06.

[145]  Masashi Shimbo,et al.  Kernels as Link Analysis Measures , 2006 .

[146]  B. Nadler,et al.  Diffusion maps, spectral clustering and reaction coordinates of dynamical systems , 2005, math/0503445.

[147]  Fabrice Rossi,et al.  A comparison between dissimilarity SOM and kernel SOM for clustering the vertices of a graph , 2007 .

[148]  F. Luccio,et al.  On the Decomposition of Networks in Minimally Interconnected Subnetworks , 1969 .

[149]  Andrew B. Kahng,et al.  Spectral Partitioning with Multiple Eigenvectors , 1999, Discret. Appl. Math..

[150]  Mikhail Belkin,et al.  Laplacian Eigenmaps for Dimensionality Reduction and Data Representation , 2003, Neural Computation.

[151]  Fan Chung,et al.  Spectral Graph Theory , 1996 .

[152]  Foster J. Provost,et al.  Classification in Networked Data: a Toolkit and a Univariate Case Study , 2007, J. Mach. Learn. Res..

[153]  Michael I. Jordan,et al.  On Spectral Clustering: Analysis and an algorithm , 2001, NIPS.

[154]  Santosh S. Vempala,et al.  On clusterings-good, bad and spectral , 2000, Proceedings 41st Annual Symposium on Foundations of Computer Science.