An Adaptive File Distribution Algorithm for Wide Area Network

This paper describes a data distribution algorithm suitable for copying large files to many nodes in multiple clusters in wide-area networks. It is a self-organizing algorithm that achieves pipeline transfers, fault tolerance, scalability, and an efficient route selection. It works in the presence of today's typical network restrictions such as firewalls and Network Address Translations, making it suitable in wide-area setting. Experimental results indicate our algorithm is able to automatically build a transfer route close to the optimal. Propagation of a 300MB file from one root node to over 150 nodes takes about 1.5 times as long as the best time obtained by the manually optimized transfer route.

[1]  Anujan Varma,et al.  Distributed algorithms for multicast path setup in data networks , 1996, TNET.

[2]  Mark Handley,et al.  A scalable content-addressable network , 2001, SIGCOMM '01.

[3]  Kirk L. Johnson,et al.  Overcast: reliable multicasting with on overlay network , 2000, OSDI.

[4]  Fred Douglis,et al.  CDN brokering , 2002, Comput. Commun..

[5]  Dinesh C. Verma,et al.  ALMI: An Application Level Multicast Infrastructure , 2001, USITS.

[6]  Srinivasan Seshan,et al.  Enabling conferencing applications on the internet using an overlay muilticast architecture , 2001, SIGCOMM 2001.

[7]  Henri Casanova,et al.  Clustering hosts in P2P and global computing platforms , 2003, CCGrid 2003. 3rd IEEE/ACM International Symposium on Cluster Computing and the Grid, 2003. Proceedings..

[8]  Ben Y. Zhao,et al.  Bayeux: an architecture for scalable and fault-tolerant wide-area data dissemination , 2001, NOSSDAV '01.

[9]  Mark Handley,et al.  Application-Level Multicast Using Content-Addressable Networks , 2001, Networked Group Communication.

[10]  Sandy Irani,et al.  Cost-Aware WWW Proxy Caching Algorithms , 1997, USENIX Symposium on Internet Technologies and Systems.

[11]  Ben Y. Zhao,et al.  An Infrastructure for Fault-tolerant Wide-area Location and Routing , 2001 .

[12]  Lili Qiu,et al.  On the placement of Web server replicas , 2001, Proceedings IEEE INFOCOM 2001. Conference on Computer Communications. Twentieth Annual Joint Conference of the IEEE Computer and Communications Society (Cat. No.01CH37213).

[13]  Lisa Higham,et al.  Self-Stabilizing Minimum Spanning Tree Construction on Message-Passing Networks , 2001, DISC.