Identifying the Most Recent Heavy Hitters in Large-Scale Streams Using Block-wise Counting

Identifying the most recent heavy hitters, i.e., finding the items with the highest appearances in a high speed data stream is a fundamental problem in real-time stream processing. The requirement of real-time stream applications raises significant challenges to this problem in terms of the processing latency, the space usage and the precision. Traditional schemes leverage the sliding windows based design which is hard to support both high precision, and low space usage of heavy hitters identification. In this work, we propose a novel Block-wise Counting scheme, which can partition the streams into tiny blocks to support high precision and low latency of heavy hitters identification with low space cost. The experiment results show that our scheme significantly improves the identification precision by 65% and reduces the processing latency by 87% compared to state-of-the-art designs.