Dynamic Load Balancing with Multiple Hash Functions in Structured P2P Systems

P2P systems can be used to form a low latency decentralized data delivery system. Structured P2P systems provide both low latency and excellent load balance with uniform query and data distributions, however, in the real application environment, requests distributions are often skewed and follow a Zipf law, some objects will become hotspots, individual nodes are easily overloaded, resulting in poor global performance and lost messages. This paper describes a novel approach with multiple hash functions to replicate the hotspots in a series of different nodes to distribute the high load evenly, and it can increase or decrease the replicas dynamically. At the same time it maintains low access latencies and good load balancing even under highly skewed demand. Results from performance evaluation demonstrate the effectiveness of our approach.

[1]  Antony I. T. Rowstron,et al.  Pastry: Scalable, Decentralized Object Location, and Routing for Large-Scale Peer-to-Peer Systems , 2001, Middleware.

[2]  Li Fan,et al.  Web caching and Zipf-like distributions: evidence and implications , 1999, IEEE INFOCOM '99. Conference on Computer Communications. Proceedings. Eighteenth Annual Joint Conference of the IEEE Computer and Communications Societies. The Future is Now (Cat. No.99CH36320).

[3]  Henning Schulzrinne,et al.  Using an External DHT as a SIP Location Service , 2006 .

[4]  Venkata N. Padmanabhan,et al.  The content and access dynamics of a busy web site: findings and implicatins , 2000, SIGCOMM.

[5]  Ben Y. Zhao,et al.  An Infrastructure for Fault-tolerant Wide-area Location and Routing , 2001 .

[6]  Mark Handley,et al.  A scalable content-addressable network , 2001, SIGCOMM '01.

[7]  Antony I. T. Rowstron,et al.  Storage management and caching in PAST, a large-scale, persistent peer-to-peer storage utility , 2001, SOSP.

[8]  Ashish Gupta,et al.  Distributed Popularity Indices , 2005 .

[9]  David R. Karger,et al.  Chord: A scalable peer-to-peer lookup service for internet applications , 2001, SIGCOMM '01.

[10]  Peter Druschel,et al.  Pastry: Scalable, distributed object location and routing for large-scale peer-to- , 2001 .

[11]  Zhiyong Xu,et al.  Effective Load Balancing in P2P Systems , 2006, Sixth IEEE International Symposium on Cluster Computing and the Grid (CCGRID'06).

[12]  Jeffrey Considine,et al.  Simple Load Balancing for Distributed Hash Tables , 2003, IPTPS.

[13]  Stefan Saroiu,et al.  A Measurement Study of Peer-to-Peer File Sharing Systems , 2001 .

[14]  Shigang Chen,et al.  Load balancing with multiple hash functions in peer-to-peer networks , 2006, 12th International Conference on Parallel and Distributed Systems - (ICPADS'06).

[15]  Miguel Castro,et al.  SCRIBE: The Design of a Large-Scale Event Notification Infrastructure , 2001, Networked Group Communication.

[16]  Jeffrey Considine,et al.  Geometric generalizations of the power of two choices , 2004, SPAA '04.

[17]  Ben Y. Zhao,et al.  Tapestry: An Infrastructure for Fault-tolerant Wide-area Location and , 2001 .

[18]  David R. Karger,et al.  Wide-area cooperative storage with CFS , 2001, SOSP.

[19]  Rajmohan Rajaraman,et al.  Accessing Nearby Copies of Replicated Objects in a Distributed Environment , 1999, Theory of Computing Systems.