Exploring Computation Locality of Graph Mining Algorithms on MapReduce

Previous implementations of graph mining algorithms on MapReduce ignore the characteristic of locality in distributed systems. For distributed systems, locality means the operations take place in local computing nodes without the communication with remote computing nodes. In this paper we present LI-MR (Local Iteration MapReduce) framework to improve a class of graph operators which can be described by repeated matrix-vector multiplications. LI-MR considers locality of sub graphs and adopts coarse granularity of communication unit for MapReduce. In particular, for sub graphs, only partial operations need synchronization. We propose a method to implement random data access on Hadoop by outputting the results to HBase. With the support of range query provided by HBase, LI-MR allows sub graphs to fulfil computation task with enough information in main memory. Because the locality feature of sub graphs, the info for the computation is limited. In this way, LI-MR framework combines in-memory computation with MapReduce model for graph algorithms.