Measurements and analytics of wide-area file transfers over dedicated connections

Distributed scientific and big-data computations are becoming increasingly dependent on access to remote files. Wide-area file transfers are supported by two basic schemes: (i) application-level tools, such as GridFTP, that provide transport services between file systems housed at geographically separated sites, and (ii) file systems mounted over wide-area networks, using mechanisms such as LNet routers that make them transparently available. In both cases, the file transfer performance critically depends on the configuration consisting of host, file, IO, and disk subsystems, which are complex by themselves, as well as on their complex compositions implemented using buffers and IO-network data transitions. We present extensive file transfer rate measurements collected over dedicated 10 Gbps connections with 0-366 ms round-trip times, using GridFTP and XDD file transfer tools, and Lustre file system extended over wide-area networks using LNet routers. Our test configurations are composed of: three types of host systems; XFS, Lustre, and ext3 file systems; and Ethernet and SONET wide-area connections. We present analytics based on the convexity-concavity of throughput profiles which provide insights into throughput and its superior or inferior trend compared to linear interpolations. We propose the utilization-concavity coefficient, a scalar metric that characterizes the overall performance of any file transfer method consisting of specific configuration and scheme. Our results enable performance optimizations by highlighting the significant roles of (i) buffer sizes and parallelism in GridFTP and XDD, and (ii) buffer utilization and credit mechanism in LNet routers.

[1]  Robert L. Grossman,et al.  UDT: UDP-based data transfer for high-speed wide area networks , 2007, Comput. Networks.

[2]  William E. Allcock,et al.  The Globus Striped GridFTP Framework and Server , 2005, ACM/IEEE SC 2005 Conference (SC'05).

[3]  Scott Michael,et al.  Demonstrating Lustre over a 100Gbps wide area network of 3,500km , 2012, 2012 International Conference for High Performance Computing, Networking, Storage and Analysis.

[4]  Stephen W. Poole,et al.  A technique for moving large data sets over high-performance long distance networks , 2011, 2011 IEEE 27th Symposium on Mass Storage Systems and Technologies (MSST).

[5]  Neena Imam,et al.  Wide-area lustre file system using LNet routers , 2018, 2018 Annual IEEE International Systems Conference (SysCon).

[6]  Tom Kelly,et al.  Scalable TCP: improving performance in highspeed wide area networks , 2003, CCRV.

[7]  Neena Imam,et al.  On Analytics of File Transfer Rates over Dedicated Wide-Area Connections , 2017, 2017 IEEE 13th International Conference on e-Science (e-Science).

[8]  Ian T. Foster,et al.  Globus: a Metacomputing Infrastructure Toolkit , 1997, Int. J. High Perform. Comput. Appl..

[9]  Scott Michael,et al.  A study of lustre networking over a 100 gigabit wide area network with 50 milliseconds of latency , 2012, DIDC '12.

[10]  Peter L. Bartlett,et al.  Neural Network Learning - Theoretical Foundations , 1999 .

[11]  Donald F. Towsley,et al.  TCP Throughput Profiles Using Measurements over Dedicated Connections , 2017, HPDC.

[12]  Min Zhu,et al.  B4: experience with a globally-deployed software defined wan , 2013, SIGCOMM.

[13]  Donald F. Towsley,et al.  Sustained Wide-Area TCP Memory Transfers over Dedicated Connections , 2015, 2015 IEEE 17th International Conference on High Performance Computing and Communications, 2015 IEEE 7th International Symposium on Cyberspace Safety and Security, and 2015 IEEE 12th International Conference on Embedded Software and Systems.

[14]  Qiang Liu,et al.  On Concavity and Utilization Analytics of Wide-Area Network Transport Protocols , 2018, 2018 IEEE 20th International Conference on High Performance Computing and Communications; IEEE 16th International Conference on Smart City; IEEE 4th International Conference on Data Science and Systems (HPCC/SmartCity/DSS).

[15]  Chase Qishi Wu,et al.  Experimental Analysis of File Transfer Rates over Wide-Area Dedicated Connections , 2016, 2016 IEEE 18th International Conference on High Performance Computing and Communications; IEEE 14th International Conference on Smart City; IEEE 2nd International Conference on Data Science and Systems (HPCC/SmartCity/DSS).

[16]  Douglas J. Leith,et al.  H-TCP : TCP for high-speed and long-distance networks , 2004 .

[17]  Chase Qishi Wu,et al.  Experiments and Analyses of Data Transfers over Wide-Area Dedicated Connections , 2017, 2017 26th International Conference on Computer Communication and Networks (ICCCN).