End-to-End Data-Flow Parallelism for Throughput Optimization in High-Speed Networks

The increase in the data produced by large-scale scientific applications necessitates innovative solutions for efficient transfer of data. Although the current optical networking technology reached theoretical speeds of 100 Gbps, applications still suffer from inefficient transport protocols and bottlenecks on the end-systems (e.g. disk, CPU, NIC). High-performance systems provide us with parallel disks, processors and network interfaces. However the lack of orchestration of these end-system resources with the available network capacity results in underutilization of the network bandwidth. In this study, a model and two algorithms that use ‘end-to-end data-flow parallelism’ to optimize the use of network and end-system resources are proposed. This is achieved by using multiple parallel streams over the network; and multiple parallel disks and CPUs at the end systems. Our model predicts the optimal number of streams and disk/CPU stripes that maximizes the data transfer speed for any setting. Our algorithms use GridFTP parallel samplings and calculate the optimal level of parallelism based on our prediction model. The experiments conducted by using actual GridFTP transfers show that the predictions performed by our model and algorithms provide close-to-optimal performances with negligible overhead and use minimal number of resources. The end-to-end data transfer throughput is improved dramatically in existence of end-system bottlenecks compared to the non-optimized transfers.