SpiderCast: a scalable interest-aware overlay for topic-based pub/sub communication

We introduce SpiderCast, a distributed protocol for constructing scalable churn-resistant overlay topologies for supporting decentralized topic-based pub/sub communication. SpiderCast is designed to effectively tread the balance between average overlay degree and communication cost of event dissemination. It employs a novel coverage-optimizing heuristic in which the nodes utilize partial subscription views (provided by a decentralized membership service) to reduce the average node degree while guaranteeing (with high probability) that the events posted on each topic can be routed solely through the nodes interested in this topic (in other words, the overlay is topic-connected). SpiderCast is unique in maintaining an overlay topology that scales well with the average number of topics a node is subscribed to, assuming the subscriptions are correlated insofar as found in most typical workloads. Furthermore, the degree grows logarithmically in the total number of topics, and slowly decreases as the number of nodes increases. We show experimentally that, for many practical work-loads, the SpiderCast overlays are both topic-connected and have a low per-topic diameter while requiring each node to maintain a low average number of connections. These properties are satisfied even in very large settings involving up to 10,000 nodes, 1,000 topics, and 70 subscriptions per-node, and under high churn rates. In addition, our results demonstrate that, in a large setting, the average node degree in SpiderCast is at least 45% smaller than in other overlays typically used to support decentralized pub/sub communication (such as e.g., similarity-based, rings-based, and random overlays).

[1]  Anne-Marie Kermarrec,et al.  Lightweight probabilistic broadcast , 2003, TOCS.

[2]  Emin Gün Sirer,et al.  Client behavior and feed characteristics of RSS, a publish-subscribe system for web micronews , 2005, IMC '05.

[3]  Weixun Li.  Semantic Peer-to-Peer Overlays for Publish / Subscribe Networks , 2007 .

[4]  Kirk L. Johnson,et al.  Overcast: reliable multicasting with on overlay network , 2000, OSDI.

[5]  Kenneth P. Birman,et al.  Bimodal multicast , 1999, TOCS.

[6]  Ajoy Kumar Datta,et al.  A Semantic Overlay for Self- Peer-to-Peer Publish/Subscribe , 2006, 26th IEEE International Conference on Distributed Computing Systems (ICDCS'06).

[7]  N. Wormald,et al.  Models of the , 2010 .

[8]  Saurabh Bagchi,et al.  Exactly-once delivery in a content-based publish-subscribe system , 2002, Proceedings International Conference on Dependable Systems and Networks.

[9]  Emin Gün Sirer,et al.  Corona: A High Performance Publish-Subscribe System for the World Wide Web , 2006, NSDI.

[10]  Anne-Marie Kermarrec,et al.  GosSkip: a Gossip-based Structured Overlay Network for Efficient Content-based Filtering , 2004 .

[11]  Amin Vahdat,et al.  Bullet: high bandwidth data dissemination using an overlay mesh , 2003, SOSP '03.

[12]  Miguel Castro,et al.  Scribe: a large-scale and decentralized application-level multicast infrastructure , 2002, IEEE J. Sel. Areas Commun..

[13]  Anne-Marie Kermarrec,et al.  Sub-2-Sub: Self-Organizing Content-Based Publish and Subscribe for Dynamic and Large Scale Collaborative Networks , 2006 .

[14]  Yoav Tock,et al.  Hierarchical Clustering of Message Flows in a Multicast Data Dissemination System , 2005, IASTED PDCS.

[15]  Alejandro P. Buchmann,et al.  A peer-to-peer approach to content-based publish/subscribe , 2003, DEBS '03.

[16]  Peter Druschel,et al.  FeedTree: Sharing Web Micronews with Peer-to-Peer Event Notification , 2005, IPTPS.

[17]  Roberto Baldoni,et al.  Content-Based Publish-Subscribe over Structured Overlay Networks , 2005, 25th IEEE International Conference on Distributed Computing Systems (ICDCS'05).

[18]  John E. Hopcroft,et al.  Correctness of a gossip based membership protocol , 2005, PODC '05.

[19]  Anne-Marie Kermarrec,et al.  Sub-2-Sub: Self-Organizing Content-Based Publish Subscribe for Dynamic Large Scale Collaborative Networks , 2006, IPTPS.

[20]  Idit Keidar,et al.  Araneola: a scalable reliable multicast system for dynamic environments , 2004, Third IEEE International Symposium on Network Computing and Applications, 2004. (NCA 2004). Proceedings..

[21]  Chunqiang Tang,et al.  GoCast: gossip-enhanced overlay multicast for fast and dependable group communication , 2005, 2005 International Conference on Dependable Systems and Networks (DSN'05).

[22]  Scott Shenker,et al.  Making gnutella-like P2P systems scalable , 2003, SIGCOMM '03.

[23]  Patrick Th. Eugster,et al.  Data-aware multicast , 2004, International Conference on Dependable Systems and Networks, 2004.