Coflow: An Application Layer Abstraction for Cluster Networking

Cluster computing applications, whether frameworks like MapReduce and Dryad, or customized applications like search platforms and social networks, have applicationlevel requirements and higher-level abstractions to express them. Networking, however, still remains at the level of forwarding packets and balancing flows, and there exists no networking abstraction that can take advantage of the rich semantics readily available from these data parallel applications. The result is a plethora of seemingly disjoint, yet somehow connected, pieces of work to address networking challenges in these applications. We propose an application layer, data plane abstraction, coflow, that can express the requirements of (data) parallel programming models used in clusters today and makes it easier to express, reason about, and act upon these requirements.

[1]  GhemawatSanjay,et al.  The Google file system , 2003 .

[2]  Sanjay Ghemawat,et al.  MapReduce: Simplified Data Processing on Large Clusters , 2004, OSDI.

[3]  Rob Pike,et al.  Interpreting the data: Parallel analysis with Sawzall , 2005, Sci. Program..

[4]  Yuan Yu,et al.  Dryad: distributed data-parallel programs from sequential building blocks , 2007, EuroSys '07.

[5]  Nick McKeown,et al.  OpenFlow: enabling innovation in campus networks , 2008, CCRV.

[6]  Ravi Kumar,et al.  Pig latin: a not-so-foreign language for data processing , 2008, SIGMOD Conference.

[7]  Jingren Zhou,et al.  SCOPE: easy and efficient parallel processing of massive data sets , 2008, Proc. VLDB Endow..

[8]  Michael Isard,et al.  DryadLINQ: A System for General-Purpose Distributed Data-Parallel Computing Using a High-Level Language , 2008, OSDI.

[9]  Amin Vahdat,et al.  Hedera: Dynamic Flow Scheduling for Data Center Networks , 2010, NSDI.

[10]  Aart J. C. Bik,et al.  Pregel: a system for large-scale graph processing , 2010, SIGMOD Conference.

[11]  Craig Chambers,et al.  FlumeJava: easy, efficient data-parallel pipelines , 2010, PLDI '10.

[12]  Hairong Kuang,et al.  The Hadoop Distributed File System , 2010, 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST).

[13]  Joseph M. Hellerstein,et al.  MapReduce Online , 2010, NSDI.

[14]  Albert G. Greenberg,et al.  Seawall: Performance Isolation for Cloud Datacenter Networks , 2010, HotCloud.

[15]  Michael I. Jordan,et al.  Managing data transfers in computer clusters with orchestra , 2011, SIGCOMM.

[16]  Gautam Kumar,et al.  FairCloud: sharing the network in cloud computing , 2011, CCRV.

[17]  Randy H. Katz,et al.  The Datacenter Needs an Operating System , 2011, HotCloud.

[18]  A. Rowstron,et al.  Towards predictable datacenter networks , 2011, SIGCOMM.

[19]  Antony I. T. Rowstron,et al.  Better never than late: meeting deadlines in datacenter networks , 2011, SIGCOMM.

[20]  Benjamin Hindman,et al.  Dominant Resource Fairness: Fair Allocation of Multiple Resource Types , 2011, NSDI.

[21]  Randy H. Katz,et al.  Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center , 2011, NSDI.

[22]  D. Zats,et al.  DeTail: reducing the flow completion time tail in datacenter networks , 2012, CCRV.

[23]  Joseph M. Hellerstein,et al.  Distributed GraphLab: A Framework for Machine Learning in the Cloud , 2012, Proc. VLDB Endow..

[24]  Carlos Guestrin,et al.  Distributed GraphLab : A Framework for Machine Learning and Data Mining in the Cloud , 2012 .

[25]  Brighten Godfrey,et al.  Finishing flows quickly with preemptive scheduling , 2012, CCRV.

[26]  Wei Lin,et al.  Microsoft Bing Peking University , 2022 .

[27]  Michael J. Franklin,et al.  Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing , 2012, NSDI.

[28]  David A. Maltz,et al.  Surviving failures in bandwidth-constrained datacenters , 2012, CCRV.

[29]  Srikanth Kandula,et al.  Reoptimizing Data Parallel Computing , 2012, NSDI.

[30]  Scott Shenker,et al.  Discretized Streams: An Efficient and Fault-Tolerant Model for Stream Processing on Large Clusters , 2012, HotCloud.