Optimizing Deadline-Driven Bulk Data Transfers in Overlay Networks

Deadline-driven bulk data transfers frequently occur in overlay networks running data-intensive, dis- tributed workflow applications, such as grid and cloud envi- ronments. What distinguishes such transfers from other In- ternet traffic is that overlay nodes should cooperate towards the common goal of delivering all inter-dependent data timely, rather than follow individual, selfish goals. For such scenarios, we propose scheduling transfers in overlays in a globally optimal manner with respect to minimizing overall network congestion. Our optimization jointly addresses routing of transfers within the overlay and the time-domain scheduling of transfer bandwidths. We formally define and address the associated problem, the Bulk Data Routing and Transfer(BDRT) and present a linear programming-based solution to it, optimal in both routing and time domains. We additionally explore alternative approaches based on heuristic routing strategies, both oblivious and time-domain optimized. We evaluate these solutions via both PlanetLab trace-driven simulations and Internet transfer experiments, on the Intrigger wide-area grid and PlanetLab. Evaluation shows that our approach finds optimal solutions, based on estimations of job arrival times, deadlines and transfer volumes. Index Terms—congestion minimization, deadline-driven transfers, routing overlays, Internet I. INTRODUCTION

[1]  Pascale Vicat-Blanc Primet,et al.  Scheduling deadline-constrained bulk data transfers to minimize network congestion , 2007, Seventh IEEE International Symposium on Cluster Computing and the Grid (CCGrid '07).

[2]  Sujata Banerjee,et al.  Bandwidth-Aware Routing in Overlay Networks , 2008, IEEE INFOCOM 2008 - The 27th Conference on Computer Communications.

[3]  Jun Wang,et al.  TRIBLER: a social‐based peer‐to‐peer system , 2008, IPTPS.

[4]  Sujata Banerjee,et al.  S3: a scalable sensing service for monitoring large networked systems , 2006, INM '06.

[5]  MartelCharles.  Preemptive Scheduling with Release Times, Deadlines, and Due Times , 1982 .

[6]  Zheng Wang,et al.  Explicit routing algorithms for Internet traffic engineering , 1999, Proceedings Eight International Conference on Computer Communications and Networks (Cat. No.99EX370).

[7]  Ariel Orda,et al.  Networks with advance reservations: the routing perspective , 2000, Proceedings IEEE INFOCOM 2000. Conference on Computer Communications. Nineteenth Annual Joint Conference of the IEEE Computer and Communications Societies (Cat. No.00CH37064).

[8]  Satish Rao,et al.  A polynomial-time tree decomposition to minimize congestion , 2003, SPAA '03.

[9]  Richard G. Baraniuk,et al.  pathChirp: Efficient available bandwidth estimation for network paths , 2003 .

[10]  Harald Räcke,et al.  Minimizing Congestion in General Networks , 2002, FOCS.

[11]  Martin Skutella,et al.  Multicommodity flows over time: Efficient algorithms and complexity , 2003, Theor. Comput. Sci..

[12]  Hari Balakrishnan,et al.  Resilient overlay networks , 2001, SOSP.

[13]  Eli Upfal,et al.  Efficient schemes for parallel communication , 1982, PODC '82.

[14]  Charles U. Martel,et al.  Preemptive Scheduling with Release Times, Deadlines, and Due Times , 1982, JACM.

[15]  Ellen W. Zegura,et al.  Optimizing End-to-End Throughput for Data Transfers on an Overlay-TCP Path , 2005, NETWORKING.

[16]  Fan Ye,et al.  Multi-site cooperative data stream analysis , 2006, OPSR.

[17]  Paul Erdös,et al.  On random graphs, I , 1959 .

[18]  Randy H. Katz,et al.  OverQoS: An Overlay Based Architecture for Enhancing Internet QoS , 2004, NSDI.

[20]  Marcin Bienkowski,et al.  A practical algorithm for constructing oblivious routing schemes , 2003, SPAA '03.