Impact of MapReduce Policies on Job Completion Reliability and Job Energy Consumption

Recently, MapReduce has been widely employed by many companies/organizations to tackle data-intensive problems over a large-scale MapReduce cluster. To solve machine/node failure which is inevitable in a MapReduce cluster, MapReduce employs several policies, such as input-data replication and intermediate-data replication policies. To speed up job execution, MapReduce allows reduce tasks to early fetch their required intermediate data. However, the impact of these policy combinations on the job completion reliability (JCR for short) and job energy consumption (JEC for short) of a MapReduce cluster was not clear, where JCR is the reliability with which a MapReduce job can be completed by the cluster, whereas JEC is the energy consumed by the cluster to complete the job. Therefore, in this study, we analyze the JCR and JEC of a MapReduce cluster on four policy combinations (POCs for short) derived from two typical intermediate-data replication policies and two typical reduce-task assignment policies. The four POCs are further compared in extensive scenarios, which not only consider jobs at different scales with various parameters, but also give a MapReduce cluster two extreme parallel execution capabilities and diverse bandwidths. The analytical results enable MapReduce managers to comprehend how these POCs impact the JCR and JEC of a cluster and then select an appropriate POC based on the characteristics of their own MapReduce jobs and clusters.

[1]  Ravi Kumar,et al.  Pig latin: a not-so-foreign language for data processing , 2008, SIGMOD Conference.

[2]  Guanying Wang,et al.  A simulation approach to evaluating design decisions in MapReduce setups , 2009, 2009 IEEE International Symposium on Modeling, Analysis & Simulation of Computer and Telecommunication Systems.

[3]  Ekpe Okorafor,et al.  AVAILABILITY OF JOB TRACKER MACHINE IN HADOOP /M AP REDUCE ZOOKEEPER COORDINATED CLUSTERS , 2012 .

[4]  Copyright © Intel Corporation 2008 * Other names and brands may be claimed as the property of others , 2004 .

[5]  Jian Li,et al.  Scalable community detection in massive social networks using MapReduce , 2013, IBM J. Res. Dev..

[6]  J. K. Ord,et al.  Handbook of the Poisson Distribution , 1967 .

[7]  Jimmy J. Lin,et al.  Web-scale computer vision using MapReduce for multimedia data mining , 2010, MDMKDD '10.

[8]  Yuan Yu,et al.  Dryad: distributed data-parallel programs from sequential building blocks , 2007, EuroSys '07.

[9]  T. S. Eugene Ng,et al.  Understanding the effects and implications of compute node related failures in hadoop , 2012, HPDC '12.

[10]  F. Haight.  Handbook of the Poisson Distribution , 1967 .

[11]  Fang-Yie Leu,et al.  Deriving Job Completion Reliability and Job Energy Consumption for a General MapReduce Infrastructure from Single-Job Perspective , 2013, 2013 27th International Conference on Advanced Information Networking and Applications Workshops.

[12]  Ying Li,et al.  Performance under Failures of MapReduce Applications , 2011, 2011 11th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing.

[13]  Malgorzata Steinder,et al.  Performance-driven task co-scheduling for MapReduce environments , 2010, 2010 IEEE Network Operations and Management Symposium - NOMS 2010.

[14]  Luiz André Barroso,et al.  The Datacenter as a Computer: An Introduction to the Design of Warehouse-Scale Machines , 2009, The Datacenter as a Computer: An Introduction to the Design of Warehouse-Scale Machines.

[15]  Fang-Yie Leu,et al.  Analyzing job completion reliability and job energy consumption for a general MapReduce infrastructure , 2013, J. High Speed Networks.

[16]  Jignesh M. Patel,et al.  Energy management for MapReduce clusters , 2010, Proc. VLDB Endow..

[17]  Tom White,et al.  Hadoop: The Definitive Guide , 2009 .

[18]  Fang-Yie Leu,et al.  Analyzing job completion reliability and job energy consumption for a heterogeneous MapReduce cluster under different intermediate-data replication policies , 2014, The Journal of Supercomputing.

[19]  Sanjay Ghemawat,et al.  MapReduce: Simplified Data Processing on Large Clusters , 2004, OSDI.

[20]  Nan Yang,et al.  Energy Efficiency for MapReduce Workloads: An In-depth Study , 2012, ADC.

[21]  Sanjay Ghemawat,et al.  MapReduce: a flexible data processing tool , 2010, CACM.

[22]  Magdalena Balazinska,et al.  Astronomy in the Cloud: Using MapReduce for Image Co-Addition , 2010, ArXiv.

[23]  Tin Yu Wu,et al.  Towards a framework for large-scale multimedia data storage and processing on Hadoop platform , 2013, The Journal of Supercomputing.

[24]  José A. B. Fortes,et al.  CloudBLAST: Combining MapReduce and Virtualization on Distributed Resources for Bioinformatics Applications , 2008, 2008 IEEE Fourth International Conference on eScience.

[25]  Gabriel Antoniu,et al.  Optimizing intermediate data management in MapReduce computations , 2011, CloudCP '11.

[26]  Jun Huang,et al.  Massive Data MapReduce Fingerprint Discriminant Algorithm Based on Hadoop , 2012 .

[27]  Indranil Gupta,et al.  Making cloud intermediate data fault-tolerant , 2010, SoCC '10.