Experiences in building and operating ePOST, a reliable peer-to-peer application

Peer-to-peer (p2p) technology can potentially be used to build highly reliable applications without a single point of failure. However, most of the existing applications, such as file sharing or web caching, have only moderate reliability demands. Without a challenging proving ground, it remains unclear whether the full potential of p2p systems can be realized.To provide such a proving ground, we have designed, deployed and operated a p2p-based email system. We chose email because users depend on it for their daily work and therefore place high demands on the availability and reliability of the service, as well as the durability, integrity, authenticity and privacy of their email. Our system, ePOST, has been actively used by a small group of participants for over two years.In this paper, we report the problems and pitfalls we encountered in this process. We were able to address some of them by applying known principles of system design, while others turned out to be novel and fundamental, requiring us to devise new solutions. Our findings can be used to guide the design of future reliable p2p systems and provide interesting new directions for future research.

[1]  Ravi Jain,et al.  An Experimental Study of the Skype Peer-to-Peer VoIP System , 2005, IPTPS.

[2]  David Mazières,et al.  Democratizing Content Publication with Coral , 2004, NSDI.

[3]  Paul Murray,et al.  The Anubis Service , 2005 .

[4]  Antony I. T. Rowstron,et al.  Pastry: Scalable, Decentralized Object Location, and Routing for Large-Scale Peer-to-Peer Systems , 2001, Middleware.

[5]  Michael Dahlin,et al.  A scalable distributed information management system , 2004, SIGCOMM.

[6]  Burton H. Bloom,et al.  Space/time trade-offs in hash coding with allowable errors , 1970, CACM.

[7]  Roy Friedman,et al.  Trading Consistency for Availability in Distributed Systems , 1996 .

[8]  Bruce M. Maggs,et al.  The feasibility of supporting large-scale live streaming applications with dynamic application end-points , 2004, SIGCOMM.

[9]  DruschelPeter,et al.  Storage management and caching in PAST, a large-scale, persistent peer-to-peer storage utility , 2001 .

[10]  David A. Maltz,et al.  DSR: the dynamic source routing protocol for multihop wireless ad hoc networks , 2001 .

[11]  Robbert van Renesse,et al.  Scalable Management and Data Mining Using Astrolabe , 2002, IPTPS.

[12]  Geoffrey M. Voelker,et al.  Surviving Internet Catastrophes , 2005, USENIX Annual Technical Conference, General Track.

[13]  Larry L. Peterson,et al.  Reliability and Security in the CoDeeN Content Distribution Network , 2004, USENIX Annual Technical Conference, General Track.

[14]  Miguel Castro,et al.  Secure routing for structured peer-to-peer overlay networks , 2002, OSDI '02.

[15]  Krishna P. Gummadi,et al.  Improving the Reliability of Internet Paths with One-hop Source Routing , 2004, OSDI.

[16]  Margo I. Seltzer,et al.  Berkeley DB , 1999, USENIX Annual Technical Conference, FREENIX Track.

[17]  David R. Karger,et al.  Wide-area cooperative storage with CFS , 2001, SOSP.

[18]  Miguel Castro,et al.  Performance and dependability of structured peer-to-peer overlays , 2004, International Conference on Dependable Systems and Networks, 2004.

[19]  Ben Y. Zhao,et al.  Tapestry: a resilient global-scale overlay for service deployment , 2004, IEEE Journal on Selected Areas in Communications.

[20]  Mahadev Satyanarayanan,et al.  Coda: A Highly Available File System for a Distributed Workstation Environment , 1990, IEEE Trans. Computers.

[21]  Peter Druschel,et al.  Providing Administrative Control and Autonomy in Structured Peer-to-Peer Overlays , 2004, IPTPS.

[22]  Jibin Zhan,et al.  Early Experience with an Internet Broadcast System Based on Overlay Multicast , 2004, USENIX Annual Technical Conference, General Track.

[23]  Michael Walfish,et al.  Untangling the Web from DNS , 2004, NSDI.

[24]  Bruce M. Maggs,et al.  The feasibility of supporting large-scale live streaming applications with dynamic application end-points , 2004, SIGCOMM 2004.

[25]  Vern Paxson,et al.  End-to-end Internet packet dynamics , 1997, SIGCOMM '97.

[26]  Brighten Godfrey,et al.  OpenDHT: a public DHT service and its uses , 2005, SIGCOMM '05.

[27]  Johanna D. Moore,et al.  A nested transaction mechanism for LOCUS , 1983, SOSP '83.

[28]  Emin Gün Sirer,et al.  The design and implementation of a next generation name service for the internet , 2004, SIGCOMM '04.

[29]  Wei Chen,et al.  Enforcing Routing Consistency in Structured Peer-to-Peer Overlays: Should We and Could We? , 2006, IPTPS.

[30]  Scott Shenker,et al.  Complex Queries in Dht-based Peer-to-peer Networks , 2002 .

[31]  Peter Druschel,et al.  Pastry: Scalable, distributed object location and routing for large-scale peer-to- , 2001 .

[32]  Marvin Theimer,et al.  Feasibility of a serverless distributed file system deployed on an existing set of desktop PCs , 2000, SIGMETRICS '00.

[33]  Stefan Savage,et al.  Inside the Slammer Worm , 2003, IEEE Secur. Priv..

[34]  Andreas Haeberlen,et al.  Glacier: highly durable, decentralized storage despite massive correlated failures , 2005, NSDI.

[35]  Alan E. Mislove,et al.  POST: A decentralized platform for reliable collaborative applications , 2005 .

[36]  David R. Karger,et al.  Chord: A scalable peer-to-peer lookup service for internet applications , 2001, SIGCOMM '01.

[37]  Hari Balakrishnan,et al.  Resilient overlay networks , 2001, SOSP.

[38]  Ben Y. Zhao,et al.  OceanStore: an architecture for global-scale persistent storage , 2000, SIGP.

[39]  David R. Karger,et al.  OverCite: A Cooperative Digital Research Library , 2005, IPTPS.

[40]  Antony I. T. Rowstron,et al.  Storage management and caching in PAST, a large-scale, persistent peer-to-peer storage utility , 2001, SOSP.

[41]  Pierre Sens,et al.  Proceedings of Hotos Ix: the 9th Workshop on Hot Topics in Operating Systems Hotos Ix: the 9th Workshop on Hot Topics in Operating Systems Post: a Secure, Resilient, Cooperative Messaging System * , 2022 .

[42]  Srinivasan Seshan,et al.  A case for end system multicast , 2002, IEEE J. Sel. Areas Commun..

[43]  Miguel Castro,et al.  Scribe: a large-scale and decentralized application-level multicast infrastructure , 2002, IEEE J. Sel. Areas Commun..

[44]  Brian D. Noble,et al.  Proceedings of the 5th Symposium on Operating Systems Design and Implementation Pastiche: Making Backup Cheap and Easy , 2022 .

[45]  Ion Stoica,et al.  Non-Transitive Connectivity and DHTs , 2005, WORLDS.

[46]  Zhe Wang,et al.  CoDNS: Improving DNS Performance and Reliability via Cooperative Lookups , 2004, OSDI.

[47]  Scott Shenker,et al.  Spurring Adoption of DHTs with OpenHash, a Public DHT Service , 2004, IPTPS.

[48]  Ralph C. Merkle,et al.  Protocols for Public Key Cryptosystems , 1980, 1980 IEEE Symposium on Security and Privacy.

[49]  Ben Y. Zhao,et al.  An Empirical Study of Collusion Behavior in the Maze P2P File-Sharing System , 2007, 27th International Conference on Distributed Computing Systems (ICDCS '07).

[50]  Miguel Castro,et al.  Security for Structured Peer-to-peer Overlay Networks , 2004 .

[51]  John R. Douceur,et al.  The Sybil Attack , 2002, IPTPS.