Efficient Inter-Datacenter Bulk Transfers with Mixed Completion Time Objectives

Bulk transfers from one to multiple datacenters can have many different completion time objectives ranging from quickly replicating some $k$ copies to minimizing the time by which the last destination receives a full replica. We design an SDN-style wide-area traffic scheduler that optimizes different completion time objectives for various requests. The scheduler builds, for each bulk transfer, one or more multicast forwarding trees which preferentially use lightly loaded network links. Multiple multicast trees are used per bulk transfer to insulate destinations that have higher available bandwidth and can hence finish quickly from congested destinations. These decisions--how many trees to construct and which receivers to serve using a given tree--result from an optimization problem that minimizes a weighted sum of transfers' completion time objectives and their bandwidth consumption. Results from simulations and emulations on Mininet show that our scheduler, Iris, can improve different completion time objectives by about $2.5\times$.

[1]  Bobby Bhattacharjee,et al.  Scalable application layer multicast , 2002, SIGCOMM '02.

[2]  David Meyer,et al.  IANA Guidelines for IPv4 Multicast Address Assignments , 2001, RFC.

[3]  Mung Chiang,et al.  Distributed wide-area traffic management for cloud services , 2012, SIGMETRICS '12.

[4]  Donald F. Towsley,et al.  Efficient rate-controlled bulk data transfer using multiple multicast groups , 2003, TNET.

[5]  Bharat K. Bhargava,et al.  PROMISE: peer-to-peer media streaming using CollectCast , 2003, MULTIMEDIA '03.

[6]  T. S. Eugene Ng,et al.  Blast: Accelerating high-performance data analytics applications by optical multicast , 2015, 2015 IEEE Conference on Computer Communications (INFOCOM).

[7]  Chen Chen,et al.  Datacast: A Scalable and Efficient Reliable Group Data Delivery Service for Data Centers , 2012, IEEE Journal on Selected Areas in Communications.

[8]  Dan Li,et al.  RDCM: Reliable data center multicast , 2011, 2011 Proceedings IEEE INFOCOM.

[9]  Hong-Fang Yu,et al.  Deadline-Guaranteed Point-to-Multipoint Bulk Transfers in Inter-Datacenter Networks , 2018, 2018 IEEE International Conference on Communications (ICC).

[10]  Srikanth Kandula,et al.  Achieving high utilization with software-driven WAN , 2013, SIGCOMM.

[11]  Alex X. Liu,et al.  Multiple bulk data transfers scheduling among datacenters , 2014, Comput. Networks.

[12]  Fang Wang,et al.  MCTCP: Congestion-aware and robust multicast TCP in Software-Defined networks , 2016, 2016 IEEE/ACM 24th International Symposium on Quality of Service (IWQoS).

[13]  Murali S. Kodialam,et al.  Online multicast routing with bandwidth guarantees: a new approach using multicast network flow , 2003, TNET.

[14]  Alex C. Snoeren,et al.  Inside the Social Network's (Datacenter) Network , 2015, Comput. Commun. Rev..

[15]  Dana S. Richards,et al.  Steiner tree problems , 1992, Networks.

[16]  David R. Cheriton,et al.  TCP-SMO: extending TCP to support medium-scale multicast applications , 2002, Proceedings.Twenty-First Annual Joint Conference of the IEEE Computer and Communications Societies.

[17]  John P. Rula,et al.  Anycast on the Move : A Look at Mobile Anycast Performance Sarah Wassermann , 2018 .

[18]  Zongpeng Li,et al.  Youtube traffic characterization: a view from the edge , 2007, IMC '07.

[19]  Laurent Massoulié,et al.  Push-to-Peer Video-on-Demand System: Design and Evaluation , 2007, IEEE Journal on Selected Areas in Communications.

[20]  Jon Crowcroft,et al.  TCP-XM: unicast-enabled reliable multicast , 2005, Proceedings. 14th International Conference on Computer Communications and Networks, 2005. ICCCN 2005..

[21]  Dimitri Watel,et al.  A practical greedy approximation for the directed Steiner tree problem , 2014, J. Comb. Optim..

[22]  Gerald Q. Maguire,et al.  The nearest replica can be farther than you think , 2015, SoCC.

[23]  Christos Gkantsidis,et al.  Comprehensive view of a live network coding P2P system , 2006, IMC '06.

[24]  Miguel Castro,et al.  SplitStream: high-bandwidth multicast in cooperative environments , 2003, SOSP '03.

[25]  Georgios Smaragdakis,et al.  The growing complexity of content delivery networks: Challenges and implications for the Internet ecosystem , 2017 .

[26]  Srikanth Kandula,et al.  Fast and Efficient Bulk Multicasting over Dedicated Inter-Datacenter Networks , 2018, ArXiv.

[27]  Peng Xu,et al.  Efficient algorithms for scheduling multiple bulk data transfers in inter‐datacenter networks , 2014, Int. J. Commun. Syst..

[28]  Srikanth Kandula,et al.  QuickCast: Fast and Efficient Inter-Datacenter Transfers Using Forwarding Tree Cohorts , 2018, IEEE INFOCOM 2018 - IEEE Conference on Computer Communications.

[29]  Ellen W. Zegura,et al.  Multiple-channel multicast scheduling for scalable bulk-data transport , 1999, IEEE INFOCOM '99. Conference on Computer Communications. Proceedings. Eighteenth Annual Joint Conference of the IEEE Computer and Communications Societies. The Future is Now (Cat. No.99CH36320).

[30]  Ming Zhang,et al.  Guaranteeing deadlines for inter-datacenter transfers , 2015, EuroSys.

[31]  Sanjoy Paul,et al.  Centralized multicast , 1999, Proceedings. Seventh International Conference on Network Protocols.

[32]  Masato Tsuru,et al.  Delivering a File by Multipath-Multicast on OpenFlow Networks , 2013, 2013 5th International Conference on Intelligent Networking and Collaborative Systems.

[33]  De-Nian Yang,et al.  Multicast traffic engineering for software-defined networks , 2015, IEEE INFOCOM 2016 - The 35th Annual IEEE International Conference on Computer Communications.

[34]  Srikanth Kandula,et al.  Calendaring for wide area networks , 2014, SIGCOMM.

[35]  Michael Sirivianos,et al.  Inter-datacenter bulk transfers with netstitcher , 2011, SIGCOMM.

[36]  IJsbrand Wijnands,et al.  Multicast Using Bit Index Explicit Replication (BIER) , 2017, RFC.

[37]  Johan A. Pouwelse,et al.  The Bittorrent P2P File-Sharing System: Measurements and Analysis , 2005, IPTPS.

[38]  Srikanth Kandula,et al.  DCCast: Efficient Point to Multipoint Transfers Across Datacenters , 2017, HotCloud.

[39]  Ramana Rao Kompella,et al.  Hierarchy-aware distributed overlays in data centers using DC2 , 2012, 2012 Fourth International Conference on Communication Systems and Networks (COMSNETS 2012).

[40]  Min Zhu,et al.  B4: experience with a globally-deployed software defined wan , 2013, SIGCOMM.

[41]  Rob Sherwood,et al.  Slurpie: a cooperative bulk data transfer protocol , 2004, IEEE INFOCOM 2004.

[42]  Ilias Giechaskiel,et al.  Delay Tolerant Bulk Data Transfers on the Internet , 2014 .

[43]  Ke Xu,et al.  BDS: a centralized near-optimal overlay network for inter-datacenter data replication , 2018, EuroSys.

[44]  Hanji Shen,et al.  An OpenFlow-Based Dynamic Path Adjustment Algorithm for Multicast Spanning Trees , 2013, 2013 12th IEEE International Conference on Trust, Security and Privacy in Computing and Communications.

[45]  Mark Handley,et al.  The Use of Forward Error Correction (FEC) in Reliable Multicast , 2002, RFC.

[46]  Mark Handley,et al.  NACK-Oriented Reliable Multicast (NORM) Transport Protocol , 2009, RFC.

[47]  Amin Vahdat,et al.  Scalability in adaptive multi-metric overlays , 2004, 24th International Conference on Distributed Computing Systems, 2004. Proceedings..

[48]  Michael Luby,et al.  A digital fountain approach to reliable distribution of bulk data , 1998, SIGCOMM '98.

[49]  Vijay Mann,et al.  Avalanche: Data center Multicast using software defined networking , 2014, 2014 Sixth International Conference on Communication Systems and Networks (COMSNETS).

[50]  Srikanth Kandula,et al.  Calendaring for wide area networks , 2015, SIGCOMM 2015.

[51]  Cauligi S. Raghavendra,et al.  DDCCast: Meeting Point to Multipoint Transfer Deadlines Across Datacenters using ALAP Scheduling Policy , 2017, ArXiv.

[52]  Mark Watson,et al.  Asynchronous Layered Coding (ALC) Protocol Instantiation , 2010, RFC.

[53]  Stefan Schmid,et al.  DaRTree: Deadline-Aware Multicast Transfers in Reconfigurable Wide-Area Networks , 2019, 2019 IEEE/ACM 27th International Symposium on Quality of Service (IWQoS).

[54]  Masato Tsuru,et al.  One-to-Many File Transfers Using Multipath-Multicast with Coding at Source , 2016, 2016 IEEE 18th International Conference on High Performance Computing and Communications; IEEE 14th International Conference on Smart City; IEEE 2nd International Conference on Data Science and Systems (HPCC/SmartCity/DSS).

[55]  W. Art Chaovalitwongse,et al.  Efficient and scalable provisioning of always-on multicast streaming services , 2009, Comput. Networks.

[56]  De-Nian Yang,et al.  Reliable multicast routing for software-defined networks , 2015, 2015 IEEE Conference on Computer Communications (INFOCOM).

[57]  Wei Xu,et al.  Optimizing Bulk Transfers with Software-Defined Optical WAN , 2016, SIGCOMM.

[58]  Robert Karl,et al.  Holistic configuration management at Facebook , 2015, SOSP.