A Highly-Accurate and Low-Overhead Prediction Model for Transfer Throughput Optimization

An important bottleneck for data-intensive scalable computing systems is efficient utilization of the network links that connect the collaborating institutions with their remote partners, data sources, and computational sites. To alleviate this bottleneck, we propose an application-layer throughput optimization model based on parallel stream number prediction. This new model extends our two previous models (Partial C-order and Full Second-order) to achieve higher accuracy and lower overhead predictions. Our new model, called Full C-order, outperforms both of our previous models as well as the most relevant model by others, the Partial Second-order, in terms of both accuracy and efficiency. We test and compare these four models on emulated testbeds and on production environments using a wide variety of data set sizes, RTT, and bandwidth combinations. Our comprehensive experiments confirm the superiority of our new model to the other three models.

[1]  Tevfik Kosar,et al.  A Data Throughput Prediction and Optimization Service for Widely Distributed Many-Task Computing , 2011, IEEE Transactions on Parallel and Distributed Systems.

[2]  Mehmet Balman,et al.  Stork data scheduler: mitigating the data bottleneck in e-Science , 2011, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[3]  Brian Tierney,et al.  Applied techniques for high bandwidth data transfers across wide area networks , 2001 .

[4]  Tevfik Kosar,et al.  Which network measurement tool is right for you? a multidimensional comparison study , 2008, 2008 9th IEEE/ACM International Conference on Grid Computing.

[5]  Kazumi Kumazoe,et al.  Performance of high-speed transport protocols coexisting on a long distance 10-Gbps testbed network , 2007, GridNets '07.

[6]  Y. Raghu Reddy,et al.  Web100: extended TCP instrumentation for research, education and diagnosis , 2003, CCRV.

[7]  Srinivasan Seshan,et al.  TCP behavior of a busy Internet server: analysis and improvements , 1997, Proceedings. IEEE INFOCOM '98, the Conference on Computer Communications. Seventeenth Annual Joint Conference of the IEEE Computer and Communications Societies. Gateway to the 21st Century (Cat. No.98.

[8]  Mary K. Vernon,et al.  Target bandwidth sharing using endhost measures , 2007, Perform. Evaluation.

[9]  Indranil Gupta,et al.  Budget-constrained bulk data transfer via internet and shipping networks , 2011, ICAC '11.

[10]  Brian D. Noble,et al.  The end-to-end performance effects of parallel TCP sockets on a lossy wide-area network , 2002, Proceedings 16th International Parallel and Distributed Processing Symposium.

[11]  Tevfik Kosar,et al.  Network-aware end-to-end data throughput optimization , 2011, NDM '11.

[12]  Jon Crowcroft,et al.  Differentiated end-to-end Internet services using a weighted proportional fair sharing TCP , 1998, CCRV.

[13]  John S. Heidemann,et al.  Effects of ensemble-TCP , 2000, CCRV.

[14]  Eitan Altman,et al.  Parallel TCP Sockets: Simple Model, Throughput and Validation , 2006, Proceedings IEEE INFOCOM 2006. 25TH IEEE International Conference on Computer Communications.

[15]  Peter A. Dinda,et al.  Modeling and taming parallel TCP on the wide area network , 2005, 19th IEEE International Parallel and Distributed Processing Symposium.

[16]  Peter A. Dinda,et al.  Characterizing and Predicting TCP Throughput on the Wide Area Network , 2005, 25th IEEE International Conference on Distributed Computing Systems (ICDCS'05).

[17]  Miron Livny,et al.  Stork: making data placement a first class citizen in the grid , 2004, 24th International Conference on Distributed Computing Systems, 2004. Proceedings..

[18]  W. Allcock,et al.  GridFTP protocol specification , 2002 .

[19]  Simson L. Garfinkel,et al.  An Evaluation of Amazon's Grid Computing Services: EC2, S3, and SQS , 2007 .

[20]  Robert L. Grossman,et al.  PSockets: The Case for Application-level Network Striping for Data Intensive Applications using High Speed Wide Area Networks , 2000, ACM/IEEE SC 2000 Conference (SC'00).

[21]  Mehmet Balman,et al.  A new paradigm: Data-aware scheduling in grid computing , 2009, Future Gener. Comput. Syst..

[22]  Tevfik Kosar,et al.  A Data Throughput Prediction and Optimization Service for Widely Distributed Many-Task Computing , 2011, IEEE Trans. Parallel Distributed Syst..

[23]  Miron Livny,et al.  Run-time Adaptation of Grid Data Placement Jobs , 2005, Scalable Comput. Pract. Exp..

[24]  Tevfik Kosar,et al.  Prediction of Optimal Parallelism Level in Wide Area Data Transfers , 2011, IEEE Transactions on Parallel and Distributed Systems.