Dependable Publish/Subscribe Systems for Distributed Application Development

The publish/subscribe (pub/sub) communication paradigm provides an asynchronous many-to-many communication substrate between data producers and data consumers. Using pub/sub, components of a distributed application can communicate without direct knowledge of each other which facilitates loose-coupling and scalability. Therefore, providing a managed pub/sub service can reduce the development and operational effort of Internet-scaled distributed applications. In this work, we address four non-functional requirements of distributed content-based pub/sub systems that can facilitate their adoption as the basis of a dependable pub/sub service suitable for distributed application development. Firstly, we address availability of a pub/sub service during broker failure. Broker failures can cause delivery disruption and therefore, a repair mechanism is required, along with message retransmission to prevent message loss. During repair and recovery, the latency of message delivery can temporarily increase. To address this problem, we present an epidemic protocol to allow a content-based pub/sub system to keep delivering messages with low latency, while failed brokers are recovering. Using a broker similarity metric, which takes into account the content space and the overlay topology, we control and direct gossip messages around failed brokers. Based on our evaluation, our approach is able to provide a higher message delivery ratio than the deterministic alternative at high failure rates or when broker failures follow a non-uniform distribution. Secondly, we address scalability of the hop-by-hop routing mechanism utilized in such distributed pub/sub systems. In an Internet-scale pub/sub service, this routing scheme allows brokers to correctly forward messages without requiring global knowledge. However, this model causes brokers to forward publications without knowing the volume and distance of matching subscribers, which can result in inefficient resource utilization. In order to raise the scalability of the service, we introduce a popularity-based routing mechanism. We define a utilization metric to measure the impact of forwarding a publication on the overall delivery of the system. Furthermore, we propose a new publication routing algorithm that takes into account broker resources and publication popularity among subscribers. Lastly, we propose three approaches to handle unpopular publications. Based on our evaluation, using