Distributed Graph Layout with Spark

This paper presents a novel way to draw very large graphs, especially those too big to fit the memory of a single computer. This new method takes advantage of the recent progress in distributed computing, notably using the Apache MapReduce library called Spark. Our implementation of a force-directed graph drawing algorithm and the way to compute repulsive forces in MapReduce are exhibited. We demonstrate the horizontal scalability of this algorithm and show layouts computed on a Hadoop cluster with our method.

[1]  Andreas Ludwig,et al.  A Fast Adaptive Layout Algorithm for Undirected Graphs , 1994, GD.

[2]  Michael Jünger,et al.  An Experimental Comparison of Fast Algorithms for Drawing General Large Graphs , 2005, GD.

[3]  Sanjay Ghemawat,et al.  MapReduce: Simplified Data Processing on Large Clusters , 2004, OSDI.

[4]  Hairong Kuang,et al.  The Hadoop Distributed File System , 2010, 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST).

[5]  David Auber,et al.  Tulip - A Huge Graph Visualization Framework , 2004, Graph Drawing Software.

[6]  Satoru Kawai,et al.  An Algorithm for Drawing General Undirected Graphs , 1989, Inf. Process. Lett..

[7]  Donald. Miner,et al.  MapReduce Design Patterns: Building Effective Algorithms and Analytics for Hadoop and Other Systems , 2012 .

[8]  Scott Shenker,et al.  Spark: Cluster Computing with Working Sets , 2010, HotCloud.

[9]  Ayellet Tal,et al.  Multi-Level Graph Layout on the GPU , 2007, IEEE Transactions on Visualization and Computer Graphics.

[10]  Stephen G. Kobourov,et al.  Journal of Graph Algorithms and Applications Grip: Graph Drawing with Intelligent Placement , 2022 .

[11]  Tamara Munzner,et al.  TopoLayout: Multilevel Graph Layout by Topological Features , 2007, IEEE Transactions on Visualization and Computer Graphics.

[12]  Jure Leskovec,et al.  {SNAP Datasets}: {Stanford} Large Network Dataset Collection , 2014 .

[13]  Michael Jünger,et al.  Drawing Large Graphs with a Potential-Field-Based Multilevel Algorithm , 2004, GD.

[14]  Yves Chiricota,et al.  Improved efficiency of spring embedders: taking advantage of GPU programming , 2007 .

[15]  David Harel,et al.  Graph Drawing by High-Dimensional Embedding , 2002, J. Graph Algorithms Appl..

[16]  Edward M. Reingold,et al.  Graph drawing by force‐directed placement , 1991, Softw. Pract. Exp..

[17]  Reynold Xin,et al.  GraphX: a resilient distributed graph system on Spark , 2013, GRADES.