FastReplica: Efficient Large File Distribution Within Content Delivery Networks

In this work, we consider a large-scale distributed network of servers and a problem of content distribution across it. We propose a novel algorithm, called FastReplica, for an efficient and reliable replication of large files in the Internet environment. There are a few basic ideas exploited in FastReplica. In order to replicate a large file among n nodes (n is in the range of 10-30 nodes), the original file is partitioned into n subfiles of equal size and each subfile is transferred to a different node in the group. After that, each node propagates its subfile to the remaining nodes in the group. Thus instead of the typical replication of an entire file to n nodes by using n Internet paths, connecting the original node to the replication group, FastReplica exploits n × n Internet paths within the replication group where each path is used for transferring 1/n -th of the file. We design a scalable and reliable FastReplica algorithm which can be used for replication of large files to a large group of nodes. The new method is simple and inexpensive. It does not require any changes or modifications to the existing Internet infrastructure, and at the same time, it significantly reduces the file replication time as we demonstrate through experiments on a prototype implementation of FastReplica in a wide-area testbed.

[1]  Helen J. Wang,et al.  Distributing streaming media content using cooperative networking , 2002, NOSSDAV '02.

[2]  Jeffrey Considine,et al.  Informed content delivery across adaptive overlay networks , 2002, IEEE/ACM Transactions on Networking.

[3]  Kirk L. Johnson,et al.  Overcast: reliable multicasting with on overlay network , 2000, OSDI.

[4]  Michael Luby,et al.  A digital fountain approach to reliable distribution of bulk data , 1998, SIGCOMM '98.

[5]  Pablo Rodriguez,et al.  Bringing the Web to the Network Edge: Large Caches and Satellite Distribution , 2002, Mob. Networks Appl..

[6]  Hui Zhang,et al.  A case for end system multicast (keynote address) , 2000, SIGMETRICS '00.

[7]  Dejan Kostic,et al.  Scalability and accuracy in a large-scale network emulator , 2002, CCRV.

[8]  Tatsuya Suda,et al.  ATM and satellite distribution of multimedia educational courseware , 1996, Proceedings of ICC/SUPERCOMM '96 - International Conference on Communications.

[9]  Srinivasan Seshan,et al.  A case for end system multicast , 2002, IEEE J. Sel. Areas Commun..

[10]  Sajal K. Das,et al.  Optimizing QoS-Based Multicast Routing in Wireless Networks: A Multi-objective Genetic Algorithmic Approach , 2002, NETWORKING.

[11]  Mark Handley,et al.  Topologically-aware overlay construction and server selection , 2002, Proceedings.Twenty-First Annual Joint Conference of the IEEE Computer and Communications Societies.

[12]  Avideh Zakhor,et al.  Distributed video streaming over Internet , 2001, IS&T/SPIE Electronic Imaging.

[13]  Eric Brewer,et al.  Scattercast: an architecture for internet broadcast distribution as an infrastructure service , 2000 .

[14]  Dinesh C. Verma,et al.  ALMI: An Application Level Multicast Infrastructure , 2001, USITS.

[15]  Pablo Rodriguez,et al.  Parallel-access for mirror sites in the Internet , 2000, Proceedings IEEE INFOCOM 2000. Conference on Computer Communications. Nineteenth Annual Joint Conference of the IEEE Computer and Communications Societies (Cat. No.00CH37064).

[16]  John G. Apostolopoulos,et al.  Reliable video communication over lossy packet networks using multiple state encoding and path diversity , 2000, IS&T/SPIE Electronic Imaging.

[17]  Srinivasan Seshan,et al.  Enabling conferencing applications on the internet using an overlay muilticast architecture , 2001, SIGCOMM '01.

[18]  Srinivasan Seshan,et al.  Enabling conferencing applications on the internet using an overlay muilticast architecture , 2001, SIGCOMM 2001.

[19]  John G. Apostolopoulos,et al.  On multiple description streaming with content delivery networks , 2002, Proceedings.Twenty-First Annual Joint Conference of the IEEE Computer and Communications Societies.

[20]  Hector Garcia-Molina,et al.  Streaming Live Media over a Peer-to-Peer Network , 2001 .

[21]  Michael Mitzenmacher,et al.  Accessing multiple mirror sites in parallel: using Tornado codes to speed up downloads , 1999, IEEE INFOCOM '99. Conference on Computer Communications. Proceedings. Eighteenth Annual Joint Conference of the IEEE Computer and Communications Societies. The Future is Now (Cat. No.99CH36320).