No . 2013 : 11 Concurrent Data Structures for Efficient Streaming Aggregation

In this paper, we study the problem of streaming multiway aggregation, where large data volumes are received from multiple input streams. Multiway aggregation is a fundamental computational component in data stream management systems, requiring lowlatency and high throughput solutions. We focus on the problem of designing concurrent data structures for achieving low-latency and high throughput multiway aggregation; an issue that has been overlooked in the literature. Data structures act as articulation points between pipeline stages of streaming aggregation, supporting parallelism and balancing the work between the aggregate stages. By studying the use and limitations of existing aggregate designs and the data structures they use, we motivate the need for new ones. We present two new concurrent data structures and their lock-free linearizable implementations, supporting both order-sensitive and order-insensitive aggregate functions. We also present an extensive evaluation, based on a variety of aggregation queries on two large datasets, one with data extracted from SoundCloud, a music social network, and one with data from a smart grid metering network. In all the experiments, the proposed data structures and methods improved the aggregation performance significantly, up to one order of magnitude, in terms of both processing throughput and latency over the commonly-used techniques based on queues.