AntFarm: Efficient Content Distribution with Managed Swarms

This paper describes Antfarm, a content distribution system based on managed swarms. A managed swarm couples peer-to-peer data exchange with a coordinator that directs bandwidth allocation at each peer. Antfarm achieves high throughput by viewing content distribution as a global optimization problem, where the goal is to minimize download latencies for participants subject to bandwidth constraints and swarm dynamics. The system is based on a wire protocol that enables the Antfarm coordinator to gather information on swarm dynamics, detect misbehaving hosts, and direct the peers' allotment of upload bandwidth among multiple swarms. Antfarm's coordinator grants autonomy and local optimization opportunities to participating nodes while guiding the swarms toward an efficient allocation of resources. Extensive simulations and a PlanetLab deployment show that the system can significantly outperform centralized distribution services as well as swarming systems such as BitTorrent.

[1]  M. L. Fisher,et al.  An analysis of approximations for maximizing submodular set functions—I , 1978, Math. Program..

[2]  Stephen E. Deering,et al.  Multicast routing in internetworks and extended LANs , 1988, SIGCOMM '88.

[3]  Mark S. Manasse,et al.  The Millicent Protocols for Electronic Commerce , 1995, USENIX Workshop on Electronic Commerce.

[4]  L. Jean Camp,et al.  Token and Notational Money in Electronic Commerce , 1995, USENIX Workshop on Electronic Commerce.

[5]  Peter G. Neumann,et al.  Reliability and Security , 1995 .

[6]  Peter Wayner,et al.  Digital cash - commerce on the net , 1995 .

[7]  Peter B. Danzig,et al.  A Hierarchical Internet Object Cache , 1996, USENIX Annual Technical Conference.

[8]  David R. Karger,et al.  Consistent hashing and random trees: distributed caching protocols for relieving hot spots on the World Wide Web , 1997, STOC '97.

[9]  Michael Luby,et al.  A digital fountain approach to reliable distribution of bulk data , 1998, SIGCOMM '98.

[10]  Michael Stumm,et al.  NetCents: A Lightweight Protocol for Secure Micropayments , 1998, USENIX Workshop on Electronic Commerce.

[11]  David R. Karger,et al.  Web Caching with Consistent Hashing , 1999, Comput. Networks.

[12]  Rudolf Ahlswede,et al.  Network information flow , 2000, IEEE Trans. Inf. Theory.

[13]  Christophe Diot,et al.  Deployment issues for the IP multicast service and architecture , 2000, IEEE Netw..

[14]  G. Voelker,et al.  On the scale and performance of cooperative Web proxy caching , 2000, OPSR.

[15]  Kirk L. Johnson,et al.  Overcast: reliable multicasting with on overlay network , 2000, OSDI.

[16]  Hui Zhang,et al.  Towards global network positioning , 2001, IMW '01.

[17]  Syam Gadde,et al.  Web caching and content distribution: a view from the interior , 2001, Comput. Commun..

[18]  Krishna P. Gummadi,et al.  An analysis of Internet content delivery systems , 2002, OPSR.

[19]  Srinivasan Seshan,et al.  A case for end system multicast , 2002, IEEE J. Sel. Areas Commun..

[20]  Miguel Castro,et al.  Secure routing for structured peer-to-peer overlay networks , 2002, OSDI '02.

[21]  Miguel Castro,et al.  SplitStream: high-bandwidth multicast in cooperative environments , 2003, SOSP '03.

[22]  Amin Vahdat,et al.  Bullet: high bandwidth data dissemination using an overlay mesh , 2003, SOSP '03.

[23]  Amin Vahdat,et al.  SHARP: an architecture for secure resource peering , 2003, SOSP '03.

[24]  Brian D. Noble,et al.  Samsara: honor among thieves in peer-to-peer storage , 2003, SOSP '03.

[25]  David Mazières,et al.  Rateless Codes and Big Downloads , 2003, IPTPS.

[26]  Emin Gün Sirer,et al.  KARMA : A Secure Economic Framework for Peer-to-Peer Resource Sharing , 2003 .

[27]  Dan S. Wallach,et al.  Enforcing Fair Sharing of Peer-to-Peer Resources , 2003, IPTPS.

[28]  B. Cohen,et al.  Incentives Build Robustness in Bit-Torrent , 2003 .

[29]  Larry L. Peterson,et al.  Reliability and Security in the CoDeeN Content Distribution Network , 2004, USENIX Annual Technical Conference, General Track.

[30]  John Kubiatowicz,et al.  Handling Churn in a DHT (Awarded Best Paper!) , 2004, USENIX Annual Technical Conference, General Track.

[31]  David Mazières,et al.  Democratizing Content Publication with Coral , 2004, NSDI.

[32]  Robert Tappan Morris,et al.  Vivaldi: a decentralized network coordinate system , 2004, SIGCOMM '04.

[33]  R. Srikant,et al.  Modeling and performance analysis of BitTorrent-like peer-to-peer networks , 2004, SIGCOMM '04.

[34]  David E. Culler,et al.  Operating Systems Support for Planetary-Scale Network Services , 2004, NSDI.

[35]  John Kubiatowicz,et al.  Handling churn in a DHT , 2004 .

[36]  Siddhartha Annapureddy,et al.  Shark: scaling file servers via cooperative caching , 2005, NSDI.

[37]  Johan A. Pouwelse,et al.  The Bittorrent P2P File-Sharing System: Measurements and Analysis , 2005, IPTPS.

[38]  Vinay S. Pai,et al.  Chainsaw: Eliminating Trees from Overlay Multicast , 2005, IPTPS.

[39]  Pablo Rodriguez,et al.  Should internet service providers fear peer-assisted content distribution? , 2005, IMC '05.

[40]  Chen-Nee Chuah,et al.  BASS: BitTorrent Assisted Streaming System for Video-on-Demand , 2005, 2005 IEEE 7th Workshop on Multimedia Signal Processing.

[41]  Min-You Wu,et al.  Approaches to establishing multicast overlays , 2005, 2005 IEEE International Conference on Services Computing (SCC'05) Vol-1.

[42]  Christos Gkantsidis,et al.  Network coding for large scale content distribution , 2005, Proceedings IEEE 24th Annual Joint Conference of the IEEE Computer and Communications Societies..

[43]  Amin Vahdat,et al.  Maintaining High-Bandwidth Under Dynamic Network Conditions , 2005, USENIX Annual Technical Conference, General Track.

[44]  John Kubiatowicz,et al.  ChunkCast: An Anycast Service for Large Content Distribution , 2006, IPTPS.

[45]  Michael Dahlin,et al.  BAR gossip , 2006, OSDI '06.

[46]  Stefan Schmid,et al.  Free Riding in BitTorrent is Cheap , 2006, HotNets.

[47]  David G. Andersen,et al.  An Architecture for Internet Data Transfer , 2006, NSDI.

[48]  KyoungSoo Park,et al.  Scale and Performance in the CoBlitz Large-File Distribution Service , 2006, NSDI.

[49]  Paul Francis,et al.  Chunkyspread: Multi-tree Unstructured Peer-to-Peer Multicast , 2006, IPTPS.

[50]  Siddhartha Annapureddy,et al.  Is high-quality vod feasible using P2P swarming? , 2007, WWW '07.

[51]  Michael Sirivianos,et al.  Dandelion: Cooperative Content Distribution with Robust Incentives , 2007, USENIX Annual Technical Conference.

[52]  Margo I. Seltzer,et al.  Network Coordinates in the Wild , 2007, NSDI.

[53]  Joseph Y. Halpern,et al.  Optimizing scrip systems: efficiency, crashes, hoarders, and altruists , 2007, EC '07.

[54]  Arun Venkataramani,et al.  Do incentives build robustness in bit torrent , 2007 .

[55]  Himabindu Pucha,et al.  Exploiting Similarity for Multi-Source Downloads Using File Handprints , 2007, NSDI.

[56]  Michael Sirivianos,et al.  Free-riding in BitTorrent Networks with the Large View Exploit , 2007, IPTPS.

[57]  Laurent Massoulié,et al.  ECHOS: edge capacity hosting overlays of nano data centers , 2008, CCRV.

[58]  Thomas E. Anderson,et al.  One Hop Reputations for Peer to Peer File Sharing Workloads , 2008, NSDI.

[59]  Fabián E. Bustamante,et al.  Taming the torrent: a practical approach to reducing cross-isp traffic in peer-to-peer systems , 2008, SIGCOMM '08.