Mimir: Memory-Efficient and Scalable MapReduce for Large Supercomputing Systems

In this paper we present Mimir, a new implementation of MapReduce over MPI. Mimir inherits the core principles of existing MapReduce frameworks, such as MR-MPI, while redesigning the execution model to incorporate a number of sophisticated optimization techniques that achieve similar or better performance with significant reduction in the amount of memory used. Consequently, Mimir allows significantly larger problems to be executed in memory, achieving large performance gains. We evaluate Mimir with three benchmarks on two highend platforms to demonstrate its superiority compared with that of other frameworks.

[1]  Trilce Estrada,et al.  A scalable and accurate method for classifying protein-ligand binding geometries using a MapReduce approach , 2012, Comput. Biol. Medicine.

[2]  Andrey Tovchigrechko,et al.  Parallelizing BLAST and SOM Algorithms with MapReduce-MPI Library , 2011, 2011 IEEE International Symposium on Parallel and Distributed Processing Workshops and Phd Forum.

[3]  Sanjay Ghemawat,et al.  MapReduce: Simplified Data Processing on Large Clusters , 2004, OSDI.

[4]  Bingsheng He,et al.  MrPhi: An Optimized MapReduce Framework on Intel Xeon Phi Coprocessors , 2015, IEEE Transactions on Parallel and Distributed Systems.

[5]  Torsten Hoefler,et al.  Towards Efficient MapReduce Using MPI , 2009, PVM/MPI.

[6]  Naga K. Govindaraju,et al.  Mars: A MapReduce Framework on graphics processors , 2008, 2008 International Conference on Parallel Architectures and Compilation Techniques (PACT).

[7]  Xian-He Sun,et al.  PortHadoop: Support direct HPC data processing in Hadoop , 2015, 2015 IEEE International Conference on Big Data (Big Data).

[8]  Seyong Lee,et al.  PUMA: Purdue MapReduce Benchmarks Suite , 2012 .

[9]  Steven J. Plimpton,et al.  MapReduce in MPI for Large-scale graph algorithms , 2011, Parallel Comput..

[10]  Dhabaleswar K. Panda,et al.  High-Performance Design of YARN MapReduce on Modern HPC Clusters with Lustre and RDMA , 2015, 2015 IEEE International Parallel and Distributed Processing Symposium.

[11]  Stéphane Marchand-Maillet,et al.  MRO-MPI: MapReduce overlapping using MPI and an optimized data exchange policy , 2013, Parallel Comput..

[12]  Christoforos E. Kozyrakis,et al.  Phoenix rebirth: Scalable MapReduce on a large-scale shared-memory system , 2009, 2009 IEEE International Symposium on Workload Characterization (IISWC).

[13]  Justin Talbot,et al.  Phoenix++: modular MapReduce for shared-memory systems , 2011, MapReduce '11.

[14]  Trilce Estrada,et al.  On Efficiently Capturing Scientific Properties in Distributed Big Data without Moving the Data: A Case Study in Distributed Structural Biology Using MapReduce , 2013, 2013 IEEE 16th International Conference on Computational Science and Engineering.

[15]  Tom White,et al.  Hadoop: The Definitive Guide , 2009 .

[16]  Allen D. Malony,et al.  Scaling Spark on HPC Systems , 2016, HPDC.

[17]  Pavan Balaji,et al.  Fault tolerant MapReduce-MPI for HPC clusters , 2015, SC15: International Conference for High Performance Computing, Networking, Storage and Analysis.

[18]  Brian W. Barrett,et al.  Introducing the Graph 500 , 2010 .

[19]  Yi Wang,et al.  Smart: a MapReduce-like framework for in-situ scientific analytics , 2015, SC15: International Conference for High Performance Computing, Networking, Storage and Analysis.

[20]  Zhiwei Xu,et al.  DataMPI: Extending MPI to Hadoop-Like Big Data Computing , 2014, 2014 IEEE 28th International Parallel and Distributed Processing Symposium.

[21]  Dhabaleswar K. Panda,et al.  Accelerating Spark with RDMA for Big Data Processing: Early Experiences , 2014, 2014 IEEE 22nd Annual Symposium on High-Performance Interconnects.

[22]  Motohiko Matsuda,et al.  K MapReduce: A scalable tool for data-processing and search/ensemble applications on large-scale supercomputers , 2013, 2013 IEEE International Conference on Cluster Computing (CLUSTER).

[23]  Scott Shenker,et al.  Spark: Cluster Computing with Working Sets , 2010, HotCloud.

[24]  Teng Wang,et al.  Characterization and Optimization of Memory-Resident MapReduce on HPC Systems , 2014, 2014 IEEE 28th International Parallel and Distributed Processing Symposium.