Optimal reducer placement to minimize data transfer in MapReduce-style processing

MapReduce-style processing has become the standard for large-scale distributed platforms, with Hadoop being the most popular implementation. While Hadoop achieves parallelism by scheduling Map and Reduce tasks across the available nodes, it incurs communication overhead in the Shuffle stage which sends intermediate results from mappers to reducers. The problem we solve is as follows: given a collection of mapper outputs (intermediate key-value pairs) and a partitioning of this collection among the reducers, which node should each reducer run on to minimize data transfer? We reduce two natural formulations of this problem to optimization problems for which polynomial solutions exist. We show that our techniques can cut communication costs by 50 percent or more compared to Hadoop's default reducer placement, which leads to lower network utilization and faster MapReduce job runtimes.

[1]  Garret Swart,et al.  Balancing reducer skew in MapReduce workloads using progressive sampling , 2012, SoCC '12.

[2]  Nikolaus Augsten,et al.  Load Balancing in MapReduce Based on Scalable Cardinality Estimates , 2012, 2012 IEEE 28th International Conference on Data Engineering.

[3]  Rainer E. Burkard,et al.  Linear Assignment Problems and Extensions , 1999, Handbook of Combinatorial Optimization.

[4]  Yuan Xue,et al.  Scalable load balancing for mapreduce-based record linkage , 2013, 2013 IEEE 32nd International Performance Computing and Communications Conference (IPCCC).

[5]  D. Janaki Ram,et al.  Chisel++: handling partitioning skew in MapReduce framework using efficient range partitioning technique , 2014, DIDC '14.

[6]  Robert E. Tarjan,et al.  Almost-optimum speed-ups of algorithms for bipartite matching and related problems , 1988, STOC '88.

[7]  E.L. Lawler,et al.  Optimization and Approximation in Deterministic Sequencing and Scheduling: a Survey , 1977 .

[8]  Yuan Xue,et al.  Scalable and robust key group size estimation for reducer load balancing in MapReduce , 2013, 2013 IEEE International Conference on Big Data.

[9]  NIDHI TIWARI,et al.  Classification Framework of MapReduce Scheduling Algorithms , 2015, ACM Comput. Surv..

[10]  Andrew V. Goldberg,et al.  Quincy: fair scheduling for distributed computing clusters , 2009, SOSP '09.

[11]  Richard M. Karp,et al.  A n^5/2 Algorithm for Maximum Matchings in Bipartite Graphs , 1971, SWAT.

[12]  Funda Ergün,et al.  Online load balancing for MapReduce with skewed data input , 2014, IEEE INFOCOM 2014 - IEEE Conference on Computer Communications.

[13]  Feifei Li,et al.  Building Wavelet Histograms on Large Data in MapReduce , 2011, Proc. VLDB Endow..

[14]  Geoffrey C. Fox,et al.  Investigation of Data Locality in MapReduce , 2012, 2012 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (ccgrid 2012).

[15]  Scott Shenker,et al.  Delay scheduling: a simple technique for achieving locality and fairness in cluster scheduling , 2010, EuroSys '10.

[16]  Ulrich Derigs,et al.  An augmenting path method for solving Linear Bottleneck Assignment problems , 1978, Computing.

[17]  Fang Dong,et al.  BAR: An Efficient Data Locality Driven Task Scheduling Algorithm for Cloud Computing , 2011, 2011 11th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing.

[18]  Mohammad Hammoud,et al.  Center-of-Gravity Reduce Task Scheduling to Lower MapReduce Network Traffic , 2012, 2012 IEEE Fifth International Conference on Cloud Computing.

[19]  Chen He,et al.  Matchmaking: A New MapReduce Scheduling Technique , 2011, 2011 IEEE Third International Conference on Cloud Computing Technology and Science.

[20]  Robert E. Tarjan,et al.  Faster Scaling Algorithms for Network Problems , 1989, SIAM J. Comput..

[21]  Diksha Verma,et al.  Quincy: Fair Scheduling for Distributed Computing Clusters , 2014 .

[22]  Michael I. Jordan,et al.  Managing data transfers in computer clusters with orchestra , 2011, SIGCOMM.