Towards In-Order and Exactly-Once Delivery Using Hierarchical Distributed Message Queues

In today's world, distributed message queues are used in many systems and play different roles (e.g. content delivery, notification system and message delivery tools). It is important for the queue services to be able to deliver messages at large scales with a variety of message sizes with high concurrency. An example of a commercial state of the art distributed message queue is Amazon Simple Queuing Service (SQS). SQS is a distributed message delivery fabric that is highly scalable. It can queue unlimited number of short messages (maximum size: 256 KB) and deliver them to multiple users in parallel. In order to be able to provide such high throughput at large scales, SQS omits some of features that are provided by traditional queues. SQS does not guarantee the order of the messages, nor does it guarantee the exactly once delivery. This paper addresses these limitations through the design and implementation of HDMQ, a hierarchical distributed message queue. HDMQ consist of collection of area message nodes that can be used to store messages up to 512 KB. It utilizes a round robin local load balancer to save the message and scale across the area region accordingly. HDMQ provides replication for high reliability of messages. It also provides SQS-like APIs in order to provide compatibility with current systems that currently use SQS. We performed a detailed performance evaluation and compared HDMQ to the commonly used commercial distributed queues measuring throughput, latency and price per request. We found HDMQ to outperform SQS, Windows Azure Service bus, and Iron MQ by up to 2-15x times in throughput, 1.6-39x times in latency, and all this for 13%-80% less costs.