Decentralized Ranking in Large-Scale Overlay Networks

Modern distributed systems are often characterized by very large scale, poor reliability, and extreme dynamism of the participating nodes, with a continuous flow of nodes joining and leaving the system. In order to develop robust applications in such environments, middleware services aimed at dealing with the inherent unpredictability of the underlying networks are required. One such service is aggregation. In the aggregation problem, each node is assumed to have attributes. The task is to extract global information about these attributes and make it available to the nodes. Examples include the total free storage, the average load, or the size of the network. Efficient protocols for computing several aggregates such as average, count, and variance have already been proposed. In this paper, we consider calculating the rank of nodes, where the set of nodes has to be sorted according to a numeric attribute and each node must be informed about its own rank in the global sorting. This information has a number of applications, such as slicing. It can also be applied to calculate the median or any other percentile. We propose T-Rank, a robust and completely decentralized algorithm for solving the ranking problem with minimal assumptions. Due to the characteristics of the targeted environment, we aim for a probabilistic approach and accept minor errors in the output. We present extensive empirical results that suggest near logarithmic time complexity, scalability and robustness in different failure scenarios.

[1]  Márk Jelasity,et al.  A Modular Paradigm for Building Self-Organizing Peer-to-Peer Applications , 2003, Engineering Self-Organising Systems.

[2]  Michael B. Jones,et al.  SkipNet: A Scalable Overlay Network with Practical Locality Properties , 2003, USENIX Symposium on Internet Technologies and Systems.

[3]  Rolf Stadler,et al.  Robust monitoring of network-wide aggregates through gossiping , 2007, IEEE Transactions on Network and Service Management.

[4]  Lionel M. Ni,et al.  Reliable Distributed Sorting Through the Application-Oriented Fault Tolerance Paradigm , 1989, IEEE Trans. Parallel Distributed Syst..

[5]  Anne-Marie Kermarrec,et al.  GosSkip, an Efficient, Fault-Tolerant and Self Organizing Overlay Using Gossip-based Construction and Skip-Lists Principles , 2006, Sixth IEEE International Conference on Peer-to-Peer Computing (P2P'06).

[6]  Robbert van Renesse,et al.  Astrolabe: A robust and scalable technology for distributed system monitoring, management, and data mining , 2003, TOCS.

[7]  Rolf Stadler,et al.  Robust monitoring of network-wide aggregates through gossiping , 2009, IEEE Trans. Netw. Serv. Manag..

[8]  Florian Schintke,et al.  Peer-to-Peer Computing , 2010, Euro-Par.

[9]  Sébastien Tixeuil,et al.  A fault-tolerant distributed sorting algorithm in tree networks , 1998, 1998 IEEE International Performance, Computing and Communications Conference. Proceedings (Cat. No.98CH36191).

[10]  Márk Jelasity,et al.  T-Man: Gossip-Based Overlay Topology Management , 2005, Engineering Self-Organising Systems.

[11]  Anne-Marie Kermarrec,et al.  Ordered Slicing of Very Large-Scale Overlay Networks , 2006, Sixth IEEE International Conference on Peer-to-Peer Computing (P2P'06).

[12]  Antony I. T. Rowstron,et al.  Pastry: Scalable, Decentralized Object Location, and Routing for Large-Scale Peer-to-Peer Systems , 2001, Middleware.

[13]  Robbert van Renesse.  The Importance of Aggregation , 2003, Future Directions in Distributed Computing.

[14]  Indranil Gupta,et al.  Scalable fault-tolerant aggregation in large process groups , 2001, 2001 International Conference on Dependable Systems and Networks.

[15]  Ian T. Foster,et al.  Mapping the Gnutella Network: Properties of Large-Scale Peer-to-Peer Systems and Implications for System Design , 2002, ArXiv.

[16]  William Pugh,et al.  Skip Lists: A Probabilistic Alternative to Balanced Trees , 1989, WADS.

[17]  D. Milojicic,et al.  Peer-to-Peer Computing , 2010 .

[18]  Hamid Pirahesh,et al.  Data Cube: A Relational Aggregation Operator Generalizing Group-By, Cross-Tab, and Sub-Totals , 1996, Data Mining and Knowledge Discovery.

[19]  Lionel M. Ni,et al.  Reliable distributed sorting through the application-oriented fault tolerance paradigm , 1989, [1989] Proceedings. The 9th International Conference on Distributed Computing Systems.

[20]  David R. Karger,et al.  Building peer-to-peer systems with chord, a distributed lookup service , 2001, Proceedings Eighth Workshop on Hot Topics in Operating Systems.

[21]  Ian T. Foster,et al.  Mapping the Gnutella Network , 2002, IEEE Internet Comput..

[22]  Márk Jelasity,et al.  Gossip-based aggregation in large dynamic networks , 2005, TOCS.

[23]  Márk Jelasity,et al.  Robust aggregation protocols for large-scale overlay networks , 2004, International Conference on Dependable Systems and Networks, 2004.

[24]  Márk Jelasity,et al.  Epidemic-style proactive aggregation in large overlay networks , 2004, 24th International Conference on Distributed Computing Systems, 2004. Proceedings..

[25]  M. Jelasity,et al.  T-Man : Fast Gossip-based Construction of Large-Scale Overlay Topologies 1 , 2004 .

[26]  Robbert van Renesse.  The importance of aggregation , 2003 .