Empirical Study of Job Scheduling Algorithms in Hadoop MapReduce

Abstract Several Job scheduling algorithms have been developed for Hadoop-Map Reduce model, which vary widely in design and behavior for handling different issues such as locality of data, user share fairness, and resource awareness. This article focuses on empirically evaluating the performance of three schedulers: First In First Out (FIFO), Fair scheduler, and Capacity scheduler. To carry out the experimental evaluation, we implement our own Hadoop cluster testbed, consisting of four machines, in which one of the machines works as the master node and all four machines work as slave nodes. The experiments include variation in data sizes, use of two different data processing applications, and variation in the number of nodes used in processing. The article analyzes the performance of the job scheduling algorithms based on various relevant performance measures. The results of the experiments are evident of the performance being affected by the job scheduling parameters, the type of applications, the number of nodes in the cluster, and size of the input data.

[1]  Guanying Wang,et al.  A simulation approach to evaluating design decisions in MapReduce setups , 2009, 2009 IEEE International Symposium on Modeling, Analysis & Simulation of Computer and Telecommunication Systems.

[2]  Chen He,et al.  Matchmaking: A New MapReduce Scheduling Technique , 2011, 2011 IEEE Third International Conference on Cloud Computing Technology and Science.

[3]  Jun Wang,et al.  Supporting HPC Analytics Applications with Access Patterns Using Data Restructuring and Data-Centric Scheduling Techniques in MapReduce , 2013, IEEE Transactions on Parallel and Distributed Systems.

[4]  Xuehai Zhou,et al.  Scheduling algorithm based on prefetching in MapReduce clusters , 2016, Appl. Soft Comput..

[5]  Yon Dohn Chung,et al.  Parallel data processing with MapReduce: a survey , 2012, SGMD.

[6]  Yang Wang,et al.  Budget-Driven Scheduling Algorithms for Batches of MapReduce Jobs in Heterogeneous Clouds , 2014, IEEE Transactions on Cloud Computing.

[7]  Alan Burns,et al.  A survey of hard real-time scheduling for multiprocessor systems , 2011, CSUR.

[8]  Wei Wei,et al.  A Replication-Based Mechanism for Fault Tolerance in MapReduce Framework , 2015 .

[9]  Xu Liu,et al.  Evaluating task scheduling in hadoop-based cloud systems , 2013, 2013 IEEE International Conference on Big Data.

[10]  Xiaoping Li,et al.  Heuristics for periodical batch job scheduling in a MapReduce computing framework , 2016, Inf. Sci..

[11]  Rong Gu,et al.  SHadoop: Improving MapReduce performance by optimizing job execution mechanism in Hadoop clusters , 2014, J. Parallel Distributed Comput..

[12]  Scott Shenker,et al.  Delay scheduling: a simple technique for achieving locality and fairness in cluster scheduling , 2010, EuroSys '10.

[13]  NIDHI TIWARI,et al.  Classification Framework of MapReduce Scheduling Algorithms , 2015, ACM Comput. Surv..

[14]  Quan Chen,et al.  HAT: history-based auto-tuning MapReduce in heterogeneous environments , 2013, The Journal of Supercomputing.

[15]  Kenli Li,et al.  A MapReduce task scheduling algorithm for deadline constraints , 2013, Cluster Computing.

[16]  Manjit Kaur,et al.  Big Data Visualization tool with Advancement of Challenges , 2014 .