BLADYG: A Graph Processing Framework for Large Dynamic Graphs

Abstract Recently, distributed processing of large dynamic graphs has become very popular, especially in certain domains such as social network analysis, Web graph analysis and spatial network analysis. In this context, many distributed/parallel graph processing systems have been proposed, such as Pregel, PowerGraph, GraphLab, and Trinity. However, these systems deal only with static graphs and do not consider the issue of processing evolving and dynamic graphs. In this paper, we are considering the issues of scale and dynamism in the case of graph processing systems. We present bladyg , a graph processing framework that addresses the issue of dynamism in large-scale graphs. We present an implementation of bladyg on top of akka framework. We experimentally evaluate the performance of the proposed framework by applying it to problems such as distributed k-core decomposition and partitioning of large dynamic graphs. The experimental results show that the performance and scalability of bladyg are satisfying for large-scale dynamic graphs.

[1]  Jure Leskovec,et al.  {SNAP Datasets}: {Stanford} Large Network Dataset Collection , 2014 .

[2]  Peter Nijkamp,et al.  The Evolution of the Commuting Network in Germany: Spatial and Connectivity Patterns , 2010 .

[3]  Joseph M. Hellerstein,et al.  Distributed GraphLab: A Framework for Machine Learning in the Cloud , 2012, Proc. VLDB Endow..

[4]  Alessandro Vespignani,et al.  K-core decomposition of Internet graphs: hierarchies, self-similarity and measurement biases , 2005, Networks Heterog. Media.

[5]  Özgür Ulusoy,et al.  Distributed $k$ -Core View Materializationand Maintenance for Large Dynamic Graphs , 2014, IEEE Trans. Knowl. Data Eng..

[6]  Aart J. C. Bik,et al.  Pregel: a system for large-scale graph processing , 2010, SIGMOD Conference.

[7]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[8]  Joseph Gonzalez,et al.  PowerGraph: Distributed Graph-Parallel Computation on Natural Graphs , 2012, OSDI.

[9]  Satnam Singh.  Cluster-level Logging of Containers with Containers , 2016, ACM Queue.

[10]  Sabeur Aridhi,et al.  Distributed k-core decomposition and maintenance in large dynamic graphs , 2016, DEBS.

[11]  Vipin Kumar,et al.  A Fast and High Quality Multilevel Scheme for Partitioning Irregular Graphs , 1998, SIAM J. Sci. Comput..

[12]  Jeffrey Xu Yu,et al.  Efficient Core Maintenance in Large Dynamic Graphs , 2012, IEEE Transactions on Knowledge and Data Engineering.

[13]  Sabeur Aridhi,et al.  DynamicDFEP: A Distributed Edge Partitioning Approach for Large Dynamic Graphs , 2016, IDEAS.

[14]  Craig Chambers,et al.  FlumeJava: easy, efficient data-parallel pipelines , 2010, PLDI '10.

[15]  Dimitrios M. Thilikos,et al.  Evaluating Cooperation in Communities with the k-Core Structure , 2011, 2011 International Conference on Advances in Social Networks Analysis and Mining.

[16]  Wilfred Ng,et al.  Blogel: A Block-Centric Framework for Distributed Computation on Real-World Graphs , 2014, Proc. VLDB Endow..

[17]  Amir H. Payberah,et al.  JA-BE-JA: A Distributed Algorithm for Balanced Graph Partitioning , 2013, 2013 IEEE 7th International Conference on Self-Adaptive and Self-Organizing Systems.

[18]  Wenguang Chen,et al.  Chronos: a graph engine for temporal graph analysis , 2014, EuroSys '14.

[19]  James Cheng,et al.  Distributed Maximal Clique Computation , 2014, 2014 IEEE International Congress on Big Data.

[20]  Ben Y. Zhao,et al.  Measurement-calibrated graph models for social network experiments , 2010, WWW '10.

[21]  Shirish Tatikonda,et al.  From "Think Like a Vertex" to "Think Like a Graph" , 2013, Proc. VLDB Endow..

[22]  Sabeur Aridhi,et al.  Big Graph Mining: Frameworks and Techniques , 2016, Big Data Res..

[23]  Kun-Lung Wu,et al.  Streaming Algorithms for k-core Decomposition , 2013, Proc. VLDB Endow..

[24]  Derek Wyatt.  Akka Concurrency , 2013 .

[25]  Michael J. Franklin,et al.  Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing , 2012, NSDI.

[26]  Vladimir Batagelj,et al.  Fast algorithms for determining (generalized) core groups in social networks , 2011, Adv. Data Anal. Classif..

[27]  Reynold Xin,et al.  GraphX: a resilient distributed graph system on Spark , 2013, GRADES.

[28]  Alberto Montresor,et al.  DFEP: Distributed Funding-Based Edge Partitioning , 2015, Euro-Par.

[29]  Francesco De Pellegrini,et al.  General , 1895, The Social History of Alcohol Review.

[30]  Haixun Wang,et al.  Trinity: a distributed graph engine on a memory cloud , 2013, SIGMOD '13.

[31]  Sabeur Aridhi,et al.  BLADYG: A Novel Block-Centric Framework for the Analysis of Large Dynamic Graphs , 2016, HPGP@HPDC.