A Scalable Distributed Louvain Algorithm for Large-Scale Graph Community Detection

We present a new distributed community detection algorithm for large graphs based on the Louvain method. We exploit a distributed delegate partitioning to ensure the workload and communication balancing among processors. In addition, we design a new heuristic strategy to carefully coordinate the community constitution in a distributed environment, and ensure the convergence of the distributed clustering algorithm. Our intensive experimental study has demonstrated the scalability and the correctness of our algorithm with various large-scale real-world and synthetic graph datasets using up to 32,768 processors.

[1]  Fabio Checconi,et al.  Scalable Community Detection with the Louvain Algorithm , 2015, 2015 IEEE International Parallel and Distributed Processing Symposium.

[2]  Jianping Zeng,et al.  A study of graph partitioning schemes for parallel graph community detection , 2016, Parallel Comput..

[3]  Santo Fortunato,et al.  Community detection in graphs , 2009, ArXiv.

[4]  Nitesh V. Chawla,et al.  Market basket analysis with networks , 2011, Social Network Analysis and Mining.

[5]  J. Machta,et al.  Parallel dynamics and computational complexity of network growth models. , 2004, Physical review. E, Statistical, nonlinear, and soft matter physics.

[6]  Christos Faloutsos,et al.  R-MAT: A Recursive Model for Graph Mining , 2004, SDM.

[7]  Sebastiano Vigna,et al.  The webgraph framework I: compression techniques , 2004, WWW '04.

[8]  Elke A. Rundensteiner,et al.  Effective graph clustering for path queries in digital map databases , 1996, CIKM '96.

[9]  Albert-László Barabási,et al.  Internet: Diameter of the World-Wide Web , 1999, Nature.

[10]  Andrea Lancichinetti,et al.  Benchmarks for testing community detection algorithms on directed and weighted graphs with overlapping communities. , 2009, Physical review. E, Statistical, nonlinear, and soft matter physics.

[11]  Makoto Onizuka,et al.  Rabbit Order: Just-in-Time Parallel Reordering for Fast Graph Analysis , 2016, 2016 IEEE International Parallel and Distributed Processing Symposium (IPDPS).

[12]  M. Newman,et al.  Finding community structure in very large networks. , 2004, Physical review. E, Statistical, nonlinear, and soft matter physics.

[13]  M E J Newman,et al.  Finding and evaluating community structure in networks. , 2003, Physical review. E, Statistical, nonlinear, and soft matter physics.

[14]  Jean-Loup Guillaume,et al.  Fast unfolding of communities in large networks , 2008, 0803.0476.

[15]  David A. Bader,et al.  Parallel Community Detection for Massive Graphs , 2011, PPAM.

[16]  Nancy M. Amato,et al.  Faster Parallel Traversal of Scale Free Graphs at Extreme Scale with Vertex Delegates , 2014, SC14: International Conference for High Performance Computing, Networking, Storage and Analysis.

[17]  Anantharaman Kalyanaraman,et al.  Parallel Heuristics for Scalable Community Detection , 2014, 2014 IEEE International Parallel & Distributed Processing Symposium Workshops.

[18]  Sebastiano Vigna,et al.  UbiCrawler: a scalable fully distributed Web crawler , 2004, Softw. Pract. Exp..

[19]  George Karypis,et al.  Partitioning and Load Balancing for Emerging Parallel Applications and Architectures , 2006, Parallel Processing for Scientific Computing.

[20]  Jure Leskovec,et al.  Defining and Evaluating Network Communities Based on Ground-Truth , 2012, ICDM.

[21]  Edward T. Bullmore,et al.  Neuroinformatics Original Research Article , 2022 .

[22]  Boleslaw K. Szymanski,et al.  Overlapping community detection in networks: The state-of-the-art and comparative study , 2011, CSUR.

[23]  David Lo,et al.  Hierarchical Parallel Algorithm for Modularity-Based Community Detection Using GPUs , 2013, Euro-Par.

[24]  Steve Harenberg,et al.  Community detection in large‐scale networks: a survey and empirical evaluation , 2014 .

[25]  Sanjukta Bhowmick,et al.  A Template for Parallelizing the Louvain Method for Modularity Maximization , 2013 .

[26]  Xiaoming Liu,et al.  SLPA: Uncovering Overlapping Communities in Social Networks via a Speaker-Listener Interaction Dynamic Process , 2011, 2011 IEEE 11th International Conference on Data Mining Workshops.

[27]  Jianping Zeng,et al.  Parallel Modularity-Based Community Detection on Large-Scale Graphs , 2015, 2015 IEEE International Conference on Cluster Computing.

[28]  K. Choromanski,et al.  Scale-Free Graph with Preferential Attachment and Evolving Internal Vertex Structure , 2013 .

[29]  Michael I. Jordan,et al.  On Spectral Clustering: Analysis and an algorithm , 2001, NIPS.