OpenDHT: a public DHT service and its uses

Large-scale distributed systems are hard to deploy, and distributed hash tables (DHTs) are no exception. To lower the barriers facing DHT-based applications, we have created a public DHT service called OpenDHT. Designing a DHT that can be widely shared, both among mutually untrusting clients and among a variety of applications, poses two distinct challenges. First, there must be adequate control over storage allocation so that greedy or malicious clients do not use more than their fair share. Second, the interface to the DHT should make it easy to write simple clients, yet be sufficiently general to meet a broad spectrum of application requirements. In this paper we describe our solutions to these design challenges. We also report our early deployment experience with OpenDHT and describe the variety of applications already using the system.

[1]  John Kubiatowicz,et al.  Handling churn in a DHT , 2004 .

[2]  Ben Y. Zhao,et al.  Awarded Best Student Paper! - Pond: The OceanStore Prototype , 2003 .

[3]  Harrick M. Vin,et al.  Start-time fair queueing: a scheduling algorithm for integrated services packet switching networks , 1996, SIGCOMM '96.

[4]  John R. Douceur,et al.  The Sybil Attack , 2002, IPTPS.

[5]  Scott Shenker,et al.  Spurring Adoption of DHTs with OpenHash, a Public DHT Service , 2004, IPTPS.

[6]  Timothy Roscoe,et al.  Palimpsest: Soft-Capacity Storage for Planetary-Scale Services , 2003, HotOS.

[7]  Antony I. T. Rowstron,et al.  Storage management and caching in PAST, a large-scale, persistent peer-to-peer storage utility , 2001, SOSP.

[8]  Harrick M. Vin,et al.  Start-time fair queueing: a scheduling algorithm for integrated services packet switching networks , 1996, SIGCOMM 1996.

[9]  Pierre Sens,et al.  Proceedings of Hotos Ix: the 9th Workshop on Hot Topics in Operating Systems Hotos Ix: the 9th Workshop on Hot Topics in Operating Systems Post: a Secure, Resilient, Cooperative Messaging System * , 2022 .

[10]  Robert Tappan Morris,et al.  Designing a DHT for Low Latency and High Throughput , 2004, NSDI.

[11]  Scott Shenker,et al.  Internet indirection infrastructure , 2004, IEEE/ACM Transactions on Networking.

[12]  Ben Y. Zhao,et al.  Tapestry: a resilient global-scale overlay for service deployment , 2004, IEEE Journal on Selected Areas in Communications.

[13]  Scott Shenker,et al.  Analysis and simulation of a fair queueing algorithm , 1989, SIGCOMM 1989.

[14]  Michael Walfish,et al.  Untangling the Web from DNS , 2004, NSDI.

[15]  Emin Gün Sirer,et al.  The design and implementation of a next generation name service for the internet , 2004, SIGCOMM '04.

[16]  Ben Y. Zhao,et al.  Bayeux: an architecture for scalable and fault-tolerant wide-area data dissemination , 2001, NOSSDAV '01.

[17]  Ben Y. Zhao,et al.  Towards a Common API for Structured Peer-to-Peer Overlays , 2003, IPTPS.

[18]  Michael Walfish,et al.  Peering Peer-to-Peer Providers , 2005, IPTPS.

[19]  David E. Culler,et al.  Operating Systems Support for Planetary-Scale Network Services , 2004, NSDI.

[20]  David Mazières,et al.  Democratizing Content Publication with Coral , 2004, NSDI.

[21]  Mark Handley,et al.  Application-Level Multicast Using Content-Addressable Networks , 2001, Networked Group Communication.

[22]  Werner Vogels,et al.  Dynamo: amazon's highly available key-value store , 2007, SOSP.

[23]  David R. Karger,et al.  Diminished Chord: A Protocol for Heterogeneous Subgroup Formation in Peer-to-Peer Networks , 2004, IPTPS.

[24]  Thomer M. Gil,et al.  Ivy: a read/write peer-to-peer file system , 2002, OSDI '02.

[25]  Robert Tappan Morris,et al.  Ivy: a read/write peer-to-peer file system , 2002, OSDI '02.

[26]  Josh Cates,et al.  Robust and efficient data management for a distributed hash table , 2003 .

[27]  Robert Morris,et al.  Etna: A Fault-tolerant Algorithm for Atomic Mutable DHT Data , 2005 .

[28]  Marvin Theimer,et al.  Flexible update propagation for weakly consistent replication , 1997, SOSP.

[29]  Sriram Ramabhadran,et al.  Brief announcement: prefix hash tree , 2004, PODC '04.

[30]  David R. Karger,et al.  Wide-area cooperative storage with CFS , 2001, SOSP.

[31]  Ben Y. Zhao,et al.  Pond: The OceanStore Prototype , 2003, FAST.

[32]  Scott Shenker,et al.  Querying the Internet with PIER , 2003, VLDB.

[33]  Miguel Castro,et al.  SplitStream: high-bandwidth multicast in cooperative environments , 2003, SOSP '03.

[34]  Micah Beck,et al.  An end-to-end approach to globally scalable programmable networking , 2003, FDNA '03.