Composed sketch framework for quantiles and cardinality queries over big data streams

Quantiles and Cardinality queries are important tools to analyze statistical information from big data streams. Due to the features of the streams, such as huge volume and high velocity, it is a challenging problem to quickly provide responses for the two types of queries using constrained space over big data streams. In this paper, we propose a composed sketch framework, which can support both quantiles queries and cardinality queries over the data streams. We introduce cardinality estimators into a baseline q-digest structure and propose unified sketch merging and query processing operations. Our approach can support these two types of queries simultaneously. We conduct detailed theoretical and experimental analysis in terms of query accuracy and query response time. The analytical and experimental results show that our approach can obtain accurate estimates quicker than traditional method and system in big data streams environments, and it just produces less than 0.8‰ storage overhead in TB-scale real-world data sets.