PopSub: Improving Resource Utilization in Distributed Content-based Publish/Subscribe Systems

Distributed content-based publish/subscribe systems provide a selective, scalable, and decentralized approach to data dissemination. In a pub/sub overlay network, hop-by-hop routing allows brokers to correctly forward messages without requiring global knowledge. However, this model causes brokers to forward publications without knowing the volume and distance of matching subscribers, which can result in inefficient resource utilization. In order to raise the scalability of pub/sub, we introduce Popularity-Based Publication Routing for Content-based Pub/Sub (PopSub), which is specifically designed to raise the resource utilization efficiency. We define a utilization metric to measure the impact of forwarding a publication on the overall delivery of the system. Furthermore, we propose a new publication routing algorithm that takes into account broker resources and publication popularity among subscribers. Lastly, we propose three approaches to handle unpopular publications. Based on our evaluations, using real-world workloads and traces, PopSub is able to improve resource efficiency of the brokers by up to 62%, and reduce delivery latency by up to 57% under high load.

[1]  Stanford,et al.  Learning to Discover Social Circles in Ego Networks , 2012 .

[2]  Roberto Beraldi,et al.  TERA: topic-based event routing for peer-to-peer architectures , 2007, DEBS '07.

[3]  Roberto Baldoni,et al.  Content-Based Publish-Subscribe over Structured Overlay Networks , 2005, 25th IEEE International Conference on Distributed Computing Systems (ICDCS'05).

[4]  Miguel Castro,et al.  Scribe: a large-scale and decentralized application-level multicast infrastructure , 2002, IEEE J. Sel. Areas Commun..

[5]  Hans-Arno Jacobsen,et al.  Composite Subscriptions in Content-Based Publish/Subscribe Systems , 2005, Middleware.

[6]  Jay Kreps,et al.  Kafka : a Distributed Messaging System for Log Processing , 2011 .

[7]  Christof Fetzer,et al.  Handling Overload in Publish/Subscribe Systems , 2006, 26th IEEE International Conference on Distributed Computing Systems Workshops (ICDCSW'06).

[8]  Emin Gün Sirer,et al.  Client behavior and feed characteristics of RSS, a publish-subscribe system for web micronews , 2005, IMC '05.

[9]  Krishna P. Gummadi,et al.  King: estimating latency between arbitrary internet end hosts , 2002, IMW '02.

[10]  Patrick Th. Eugster,et al.  Atmosphere: A Universal Cross-Cloud Communication Infrastructure , 2013, Middleware.

[11]  Sérgio Duarte,et al.  Routing algorithms for content-based publish/subscribe systems , 2010, IEEE Communications Surveys & Tutorials.

[12]  Pieter Hintjens,et al.  ZeroMQ: Messaging for Many Applications , 2013 .

[13]  Hans-Arno Jacobsen,et al.  Algorithms Based on Divide and Conquer for Topic-Based Publish/Subscribe Overlay Design , 2016, IEEE/ACM Transactions on Networking.

[14]  Yoav Tock,et al.  SpiderCast: a scalable interest-aware overlay for topic-based pub/sub communication , 2007, DEBS '07.

[15]  Anne-Marie Kermarrec,et al.  Lightweight probabilistic broadcast , 2003, TOCS.

[16]  Anne-Marie Kermarrec,et al.  Lightweight probabilistic broadcast , 2001, 2001 International Conference on Dependable Systems and Networks.

[17]  Jun Wei,et al.  A New Approach for Overload Management in Content-based Publish/Subscribe , 2007, International Conference on Software Engineering Advances (ICSEA 2007).

[18]  Nalini Venkatasubramanian,et al.  GSFord: Towards a Reliable Geo-social Notification System , 2012, 2012 IEEE 31st Symposium on Reliable Distributed Systems.

[19]  Helge Parzyjegla,et al.  Reconfiguring Publish/Subscribe Overlay Topologies , 2006, 26th IEEE International Conference on Distributed Computing Systems Workshops (ICDCSW'06).

[20]  Hosung Park,et al.  What is Twitter, a social network or a news media? , 2010, WWW '10.

[21]  David S. Rosenblum,et al.  Design and evaluation of a wide-area event notification service , 2001, TOCS.

[22]  Hans-Arno Jacobsen,et al.  A Unified Approach to Routing, Covering and Merging in Publish/Subscribe Systems Based on Modified Binary Decision Diagrams , 2005, 25th IEEE International Conference on Distributed Computing Systems (ICDCS'05).

[23]  Hans-Arno Jacobsen,et al.  Planning the Transformation of Distributed Messaging Middlewares , 2014 .

[24]  Paolo Bellavista,et al.  Quality of Service in Wide Scale Publish—Subscribe Systems , 2014, IEEE Communications Surveys & Tutorials.

[25]  Reza Sherafat Kazemzadeh,et al.  Opportunistic Multipath Forwarding in Content-Based Publish/Subscribe Overlays , 2012, Middleware.

[26]  Patrick Th. Eugster,et al.  Parametric Subscriptions for Content-Based Publish/Subscribe Networks , 2010, Middleware.

[27]  Anne-Marie Kermarrec,et al.  The many faces of publish/subscribe , 2003, CSUR.

[28]  M. Newman,et al.  Why social networks are different from other types of networks. , 2003, Physical review. E, Statistical, nonlinear, and soft matter physics.

[29]  Peter R. Pietzuch,et al.  Congestion Control in a Reliable Scalable Message-Oriented Middleware , 2003, Middleware.

[30]  Dave Levin,et al.  PeerWise Discovery and Negotiation of Faster Paths , 2007, HotNets.

[31]  Luís E. T. Rodrigues,et al.  Scalable QoS-Based Event Routing in Publish-Subscribe Systems , 2005, Fourth IEEE International Symposium on Network Computing and Applications.

[32]  Alfonso Fuggetta,et al.  The JEDI Event-Based Infrastructure and Its Application to the Development of the OPSS WFMS , 2001, IEEE Trans. Software Eng..

[33]  Sheila A. McIlraith,et al.  Towards Planning the Transformation of Overlays , 2015, 2015 IEEE 35th International Conference on Distributed Computing Systems.