Efficient wide area data transfer protocols for 100 Gbps networks and beyond

Due to a number of recent technology developments, now is the right time to re-examine the use of TCP for very large data transfers. These developments include the deployment of 100 Gigabit per second (Gbps) network backbones, hosts that can easily manage 40 Gbps, and higher, data transfers, the Science DMZ model, the availability of virtual circuit technology, and wide-area Remote Direct Memory Access (RDMA) protocols. In this paper we show that RDMA works well over wide-area virtual circuits, and uses much less CPU than TCP or UDP. We also characterize the limitations of RDMA in the presence of other traffic, including competing RDMA flows. We conclude that RDMA for Science DMZ to Science DMZ transfers of massive data is a viable and desirable option for high-performance data transfer.

[1]  Pavan Balaji,et al.  Sockets vs. RDMA Interface over 10-Gigabit Networks: An In-depth Analysis of the Memory Traffic Bottleneck , 2004 .

[2]  D. Martin Swany,et al.  Evaluating High Performance Data Transfer with RDMA-based Protocols in Wide-Area Networks , 2012, 2012 IEEE 14th International Conference on High Performance Computing and Communication & 2012 IEEE 9th International Conference on Embedded Software and Systems.

[3]  Brian Tierney,et al.  Efficient data transfer protocols for big data , 2012, 2012 IEEE 8th International Conference on E-Science.

[4]  Van Jacobson,et al.  TCP Extensions for High Performance , 1992, RFC.

[5]  Pete Wyckoff,et al.  A Performance Analysis of the Ammasso RDMA Enabled Ethernet Adapter and its iWARP API , 2005, 2005 IEEE International Conference on Cluster Computing.

[6]  Balázs Sonkoly,et al.  A comprehensive TCP fairness analysis in high speed networks , 2009, Comput. Commun..

[7]  Jeffrey S. Vetter,et al.  Efficient Zero-Copy Noncontiguous I/O for Globus on InfiniBand , 2010, 2010 39th International Conference on Parallel Processing Workshops.

[8]  Dorian Kcira,et al.  Efficient LHC Data Distribution across 100Gbps Networks , 2012, 2012 SC Companion: High Performance Computing, Networking Storage and Analysis.

[9]  Robert D. Russell,et al.  A Performance Study to Guide RDMA Programming Decisions , 2012, 2012 IEEE 14th International Conference on High Performance Computing and Communication & 2012 IEEE 9th International Conference on Embedded Software and Systems.

[10]  Yuan Tian,et al.  EXIO : Enabling globus on rdma networks-a case study with InfiniBand , 2010 .

[11]  Eli Dart.  Biological and Environmental Research Network Requirements , 2014 .

[12]  Weikuan Yu,et al.  Performance of RDMA-capable storage protocols on wide-area network , 2008, 2008 3rd Petascale Data Storage Workshop.

[13]  Rajkumar Kettimuthu,et al.  A GridFTP Transport Driver for Globus XIO , 2008, PDPTA.

[14]  Robert L. Grossman,et al.  UDT: UDP-based data transfer for high-speed wide area networks , 2007, Comput. Networks.

[15]  Cheng Song,et al.  High performance TCP in ANSNET , 1994, CCRV.

[16]  Brian Tierney,et al.  Protocols for wide-area data-intensive applications: Design and performance issues , 2012, 2012 International Conference for High Performance Computing, Networking, Storage and Analysis.

[17]  Walid Dabbous,et al.  On TCP performance in a heterogeneous network: a survey , 2000, IEEE Commun. Mag..

[18]  Eli Dart,et al.  The Science DMZ: A network design pattern for data-intensive science , 2013, 2013 SC - International Conference for High Performance Computing, Networking, Storage and Analysis (SC).

[19]  Dhabaleswar K. Panda,et al.  Performance of HPC Middleware over InfiniBand WAN , 2008, 2008 37th International Conference on Parallel Processing.

[20]  Arnaldo Carvalho de Melo,et al.  The New Linux ’ perf ’ Tools , 2010 .

[21]  Jeffrey S. Vetter,et al.  Wide-area performance profiling of 10GigE and InfiniBand technologies , 2008, 2008 SC - International Conference for High Performance Computing, Networking, Storage and Analysis.

[22]  William E. Johnston,et al.  Hybrid networks: lessons learned and future challenges based on ESnet4 experience , 2011, IEEE Communications Magazine.

[23]  P ? ? ? ? ? ? ? % ? ? ? ? , 1991 .