Maiter : A Message-Passing Distributed Framework for Accumulative Iterative Computation

Myriad of machine learning and data mining algorithms require parsing data sets iteratively. These iterative algorithms have to be implemented in a distributed environment in order to scale to massive data sets. To accelerate iterative computations in a large-scale distributed environment, we identify a broad class of iterative computations that can accumulate iterative update results. Specifically, different from traditional iterative computations, which iteratively update the result based on the result from the previous iteration, accumulative iterative update accumulates the intermediate iterative update results. We prove that an accumulative update will yield the same result as its corresponding traditional iterative update. Furthermore, accumulative iterative computation can be performed asynchronously and converges much faster than traditional iterative computation. We present a general computation model to describe asynchronous accumulative iterative computation. Based on the computation model, we design and implement a messagepassing distributed framework, Maiter. We evaluate Maiter on Amazon EC2 Cloud with 100 EC2 instances. Our results show that Maiter achieves as much as 60x speedup over Hadoop for implementing iterative algorithms.