A Distributed Topic-Based Pub/Sub Method for Exhaust Data Streams towards Scalable Event-Driven Systems

Distributed pub/sub messaging has become indispensable for event-driven systems. There are methods for achieving high scalability regarding topic-based pub/sub by using structured overlay networks. However, these methods waste network resources concerning "exhaust data," which have low or no value most of the time. There are at least two problems: each publisher node continues to forward data to a relay node even if there are no subscribers, and multicast trees are constructed which are excessively large for low value data, namely having a small number of subscribers. In this paper, we formulate the requirements of overlay networks by defining a property called "strong relay-free" as an expansion of relay-free property, and propose a practical method satisfying the property by using Skip Graph. The proposed method involves publishers and subscribers composing connected sub graphs to enable detecting the absence of subscribers and autonomously adjusting the tree size. Through simulation experiments, we confirmed that the proposed method can suspend publishing adaptively, and shorten the path length on multicast trees by more than 75% under an experimental condition with 100,000 nodes. The proposed method is competent for decentralized event-driven systems with encouraging the locally produced data to be consumed locally.

[1]  William Pugh,et al.  Skip Lists: A Probabilistic Alternative to Balanced Trees , 1989, WADS.

[2]  Ben Y. Zhao,et al.  Bayeux: an architecture for scalable and fault-tolerant wide-area data dissemination , 2001, NOSSDAV '01.

[3]  Mark Handley,et al.  A scalable content-addressable network , 2001, SIGCOMM '01.

[4]  Mark Handley,et al.  Application-Level Multicast Using Content-Addressable Networks , 2001, Networked Group Communication.

[5]  Antony I. T. Rowstron,et al.  Pastry: Scalable, Decentralized Object Location, and Routing for Large-Scale Peer-to-Peer Systems , 2001, Middleware.

[6]  Miguel Castro,et al.  Scribe: a large-scale and decentralized application-level multicast infrastructure , 2002, IEEE J. Sel. Areas Commun..

[7]  Michael B. Jones,et al.  SkipNet: A Scalable Overlay Network with Practical Locality Properties , 2003, USENIX Symposium on Internet Technologies and Systems.

[8]  James Aspnes,et al.  Skip graphs , 2003, SODA '03.

[9]  Y. Charlie Hu,et al.  Borg: a hybrid protocol for scalable application-level multicast in peer-to-peer networks , 2003, NOSSDAV '03.

[10]  John Hunt,et al.  Java Message Service (JMS) , 2003 .

[11]  Anne-Marie Kermarrec,et al.  The many faces of publish/subscribe , 2003, CSUR.

[12]  Ben Y. Zhao,et al.  Tapestry: a resilient global-scale overlay for service deployment , 2004, IEEE Journal on Selected Areas in Communications.

[13]  Yoav Tock,et al.  Constructing scalable overlays for pub-sub with many topics , 2007, PODC '07.

[14]  J. Manyika.  Big data: The next frontier for innovation, competition, and productivity , 2011 .

[15]  Maarten van Steen,et al.  PolderCast: Fast, Robust, and Scalable Architecture for P2P Topic-Based Pub/Sub , 2012, Middleware.

[16]  Steve Hodges,et al.  Prototyping Connected Devices for the Internet of Things , 2013, Computer.