H-PFSP: Efficient Hybrid Parallel PFSP Protected Scheduling for MapReduce System

MapReduce provides a data-parallel computing framework, and has emerged as a popular processing model due to the simplicity of operations for big data application developers. Data processing applications from many different domains such as search and data mining are usually developed using open-source Hadoop implementation of MapReduce or self-developed MapReduce-like implementations like Dryad [1] and Ciel [2]. In cloud environments, products like Amazon's Elastic Compute Cloud (EC2) [3] provide MapReduce services as third-party multi-tenant service. Even within a company, a number of products may share the MapReduce cluster. Therefore, a fair and efficient scheduler is crucial to improve performance of submitted jobs and guarantee multi-user fairness. However, in practice, it is hard to guarantee both fairness and per-job performance, especially when jobs are scheduled without accurate estimation. We show that processor sharing (PS) type of schedulers like Fair Scheduling degrade the per-job performance in a multi-user environment. We present a new scheduling policy, Hybrid Parallel pessimistic Fair Schedule Protocol (H-PFSP), that can finish every job no later than Fair scheduler does. Unlike Fair scheduler, however, it can improve the per-job performance of MapReduce systems with relatively accurate job progress estimation.

[1]  Adam Wierman,et al.  Scheduling for today's computer systems: bridging theory and practice , 2007 .

[2]  Yuan Yu,et al.  Dryad: distributed data-parallel programs from sequential building blocks , 2007, EuroSys '07.

[3]  Sanjay Ghemawat,et al.  MapReduce: Simplified Data Processing on Large Clusters , 2004, OSDI.

[4]  Ching-Hsien Hsu,et al.  An improved partitioning mechanism for optimizing massive data analysis using MapReduce , 2013, The Journal of Supercomputing.

[5]  Peter A. Dinda,et al.  Size-based scheduling policies with inaccurate scheduling information , 2004, The IEEE Computer Society's 12th Annual International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunications Systems, 2004. (MASCOTS 2004). Proceedings..

[6]  Carey L. Williamson,et al.  Simulation evaluation of hybrid SRPT scheduling policies , 2004, The IEEE Computer Society's 12th Annual International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunications Systems, 2004. (MASCOTS 2004). Proceedings..

[7]  Eric J. Friedman,et al.  Fairness and efficiency in web server protocols , 2003, SIGMETRICS '03.

[8]  Quan Chen,et al.  SAMR: A Self-adaptive MapReduce Scheduling Algorithm in Heterogeneous Environment , 2010, 2010 10th IEEE International Conference on Computer and Information Technology.

[9]  Andrew V. Goldberg,et al.  Quincy: fair scheduling for distributed computing clusters , 2009, SOSP '09.

[10]  Steven Hand,et al.  CIEL: A Universal Execution Engine for Distributed Data-Flow Computing , 2011, NSDI.

[11]  Albert G. Greenberg,et al.  Reining in the Outliers in Map-Reduce Clusters using Mantri , 2010, OSDI.

[12]  Pietro Michiardi,et al.  Shared Cluster Scheduling : a Fair and Efficient Protocol , 2011 .

[13]  Matei Zaharia,et al.  Job Scheduling for Multi-User MapReduce Clusters , 2009 .

[14]  Randy H. Katz,et al.  Improving MapReduce Performance in Heterogeneous Environments , 2008, OSDI.

[15]  Daqiang Zhang,et al.  Searching in Internet of Things: Vision and Challenges , 2011, 2011 IEEE Ninth International Symposium on Parallel and Distributed Processing with Applications.