Diamond sketch: Accurate per-flow measurement for real IP streams

Existing sketches often have low memory efficiencies when performing per-flow measurement tasks on skewed IP streams. In this paper, we propose Diamond Sketch, a novel sketch that dynamically assigns an appropriate number of atom sketches to each flow on demand, improving the accuracy considerably while keeping a comparable speed.

[1]  Gustavo Alonso,et al.  Augmented Sketch: Faster and More Accurate Stream Processing , 2016, SIGMOD Conference.

[2]  Gaogang Xie,et al.  A Shifting Bloom Filter Framework for Set Queries , 2015, Proc. VLDB Endow..

[3]  Tong Yang,et al.  A Shifting Framework for Set Queries , 2017, IEEE/ACM Transactions on Networking.

[4]  George Varghese,et al.  New directions in traffic measurement and accounting: Focusing on the elephants, ignoring the mice , 2003, TOCS.

[5]  Tong Yang,et al.  Pyramid Sketch: a Sketch Framework for Frequency Estimation of Data Streams , 2017, Proc. VLDB Endow..

[6]  Jih-Kwon Peir,et al.  Fit a Spread Estimator in Small Memory , 2009, IEEE INFOCOM 2009.

[7]  Xenofontas A. Dimitropoulos,et al.  Probabilistic lossy counting: an efficient algorithm for finding heavy hitters , 2008, CCRV.

[8]  Graham Cormode,et al.  Sketch Techniques for Approximate Query Processing , 2010 .

[9]  Minlan Yu,et al.  FlowRadar: A Better NetFlow for Data Centers , 2016, NSDI.

[10]  David M. W. Powers,et al.  Applications and Explanations of Zipf’s Law , 1998, CoNLL.

[11]  Shigang Chen,et al.  Per-Flow Traffic Measurement Through Randomized Counter Sharing , 2012, IEEE/ACM Trans. Netw..

[12]  Graham Cormode,et al.  An improved data stream summary: the count-min sketch and its applications , 2004, J. Algorithms.

[13]  Minlan Yu,et al.  Cold Filter: A Meta-Framework for Faster and More Accurate Stream Processing , 2018, SIGMOD Conference.

[14]  Moses Charikar,et al.  Finding frequent items in data streams , 2004, Theor. Comput. Sci..