Structured overlay for heterogeneous environments: Design and evaluation of oscar

Recent years have seen advances in building large Internet-scale index structures, generally known as structured overlays. Early structured overlays realized distributed hash tables (DHTs) which are ill suited for anything but exact queries. The need to support range queries necessitates systems that can handle uneven load distributions. However such systems suffer from practical problems—including poor latency, disproportionate bandwidth usage at participating peers, or unrealistic assumptions on peers' homogeneity, in terms of available storage or bandwidth resources. In this article we consider a system that is not only able to support uneven load distributions but also to operate in heterogeneous environments, where each peer can autonomously decide how much of its resources to contribute to the system. We provide the theoretical foundations of realizing such a network and present a newly proposed system Oscar based on these principles. Oscar can construct efficient overlays given arbitrary load distributions by employing a novel scalable network sampling technique. The simulations of our system validate the theory and evaluate Oscar's performance under typical challenges, encountered in real-life large-scale networked systems, including participant heterogeneity, faults, and skewed and dynamic load-distributions. Thus the Oscar distributed index fills in an important gap in the family of structured overlays, bringing into life a practical Internet-scale index, which can play a crucial role in enabling data-oriented applications distributed over wide-area networks.

[1]  Hector Garcia-Molina,et al.  Online Balancing of Range-Partitioned Data with Applications to Peer-to-Peer Systems , 2004, VLDB.

[2]  David R. Karger,et al.  Simple Efficient Load-Balancing Algorithms for Peer-to-Peer Systems , 2004, SPAA '04.

[3]  Ali Ghodsi,et al.  Distributed k-ary System: Algorithms for Distributed Hash Tables , 2006 .

[4]  Srinivasan Seshan,et al.  Mercury: supporting scalable multi-attribute range queries , 2004, SIGCOMM 2004.

[5]  David K. Y. Yau,et al.  Small-world overlay P2P networks: Construction, management and handling of dynamic flash crowds , 2006, Comput. Networks.

[6]  Li Fan,et al.  Web caching and Zipf-like distributions: evidence and implications , 1999, IEEE INFOCOM '99. Conference on Computer Communications. Proceedings. Eighteenth Annual Joint Conference of the IEEE Computer and Communications Societies. The Future is Now (Cat. No.99CH36320).

[7]  George Kingsley Zipf,et al.  Relative Frequency as a Determinant of Phonetic Change , 1930 .

[8]  Joseph M. Hellerstein,et al.  Toward network data independence , 2003, SGMD.

[9]  Karl Aberer,et al.  Indexing Data-oriented Overlay Networks , 2005, VLDB.

[10]  David K. Y. Yau,et al.  Small-World Overlay P 2 P Networks : Construction and Handling Dynamic Flash Crowd , 2022 .

[11]  Moni Naor,et al.  Know thy neighbor's neighbor: the power of lookahead in randomized P2P networks , 2004, STOC '04.

[12]  Jonathan Kirsch,et al.  Load balancing and locality in range-queriable data structures , 2004, PODC '04.

[13]  George Giakkoupis,et al.  A scheme for load balancing in heterogenous distributed hash tables , 2005, PODC '05.

[14]  Xiaozhou Li,et al.  Active and Concurrent Topology Maintenance , 2004, DISC.

[15]  James Aspnes,et al.  Fault-tolerant routing in peer-to-peer systems , 2002, PODC '02.

[16]  Peter Druschel,et al.  Pastry: Scalable, distributed object location and routing for large-scale peer-to- , 2001 .

[17]  Antony I. T. Rowstron,et al.  Pastry: Scalable, Decentralized Object Location, and Routing for Large-Scale Peer-to-Peer Systems , 2001, Middleware.

[18]  Richard M. Karp,et al.  Load Balancing in Structured P2P Systems , 2003, IPTPS.

[19]  George Giakkoupis,et al.  On the complexity of greedy routing in ring-based peer-to-peer networks , 2007, PODC '07.

[20]  Daniel Stutzbach,et al.  Characterizing unstructured overlay topologies in modern P2P file-sharing systems , 2005 .

[21]  Lali Barrière,et al.  Efficient Routing in Networks with Long Range Contacts , 2001, DISC.

[22]  David R. Karger,et al.  Chord: A scalable peer-to-peer lookup service for internet applications , 2001, SIGCOMM '01.

[23]  James Aspnes,et al.  Fast construction of overlay networks , 2005, SPAA '05.

[24]  David R. Karger,et al.  Analysis of the evolution of peer-to-peer systems , 2002, PODC '02.

[25]  James Aspnes,et al.  Skip graphs , 2003, SODA '03.

[26]  Daniel Stutzbach,et al.  Characterizing Unstructured Overlay Topologies in Modern P2P File-Sharing Systems , 2005, IEEE/ACM Transactions on Networking.

[27]  Anne-Marie Kermarrec,et al.  GosSkip, an Efficient, Fault-Tolerant and Self Organizing Overlay Using Gossip-based Construction and Skip-Lists Principles , 2006, Sixth IEEE International Conference on Peer-to-Peer Computing (P2P'06).

[28]  Ramesh K. Sitaraman,et al.  The power of two random choices: a survey of tech-niques and results , 2001 .

[29]  Richard M. Karp,et al.  Load balancing in dynamic structured P2P systems , 2004, IEEE INFOCOM 2004.

[30]  Srinivasan Seshan,et al.  Mercury: supporting scalable multi-attribute range queries , 2004, SIGCOMM '04.

[31]  Jon M. Kleinberg,et al.  The small-world phenomenon: an algorithmic perspective , 2000, STOC '00.

[32]  Karl Aberer,et al.  P-Grid: A Self-Organizing Access Structure for P2P Information Systems , 2001, CoopIS.

[33]  Robert Morris,et al.  Chord: A scalable peer-to-peer lookup service for internet applications , 2001, SIGCOMM 2001.

[34]  W. Hoeffding.  Probability Inequalities for sums of Bounded Random Variables , 1963 .

[35]  Karl Aberer,et al.  Oscar: Small-World Overlay for Realistic Key Distributions , 2005, DBISP2P.

[36]  Gurmeet Singh Manku,et al.  Symphony: Distributed Hashing in a Small World , 2003, USENIX Symposium on Internet Technologies and Systems.

[37]  Mark Handley,et al.  A scalable content-addressable network , 2001, SIGCOMM 2001.

[38]  Douglas S. Reeves,et al.  Self-stabilizing structured ring topology P2P systems , 2005, Fifth IEEE International Conference on Peer-to-Peer Computing (P2P'05).

[39]  Karl Aberer,et al.  On Small World Graphs in Non-uniformly Distributed Key Spaces , 2005, 21st International Conference on Data Engineering Workshops (ICDEW'05).

[40]  G. Zipf,et al.  Relative Frequency as a Determinant of Phonetic Change , 1930 .

[41]  Michael B. Jones,et al.  SkipNet: A Scalable Overlay Network with Practical Locality Properties , 2003, USENIX Symposium on Internet Technologies and Systems.

[42]  David Novak,et al.  M-Chord: a scalable distributed similarity search structure , 2006, InfoScale '06.

[43]  George Kachergis,et al.  Depth of Field and Cautious-Greedy Routing in Social Networks , 2007, ISAAC.