Elastic sketch: adaptive and fast network-wide measurements

When network is undergoing problems such as congestion, scan attack, DDoS attack, etc., measurements are much more important than usual. In this case, traffic characteristics including available bandwidth, packet rate, and flow size distribution vary drastically, significantly degrading the performance of measurements. To address this issue, we propose the Elastic sketch. It is adaptive to currently traffic characteristics. Besides, it is generic to measurement tasks and platforms. We implement the Elastic sketch on six platforms: P4, FPGA, GPU, CPU, multi-core CPU, and OVS, to process six typical measurement tasks. Experimental results and theoretical analysis show that the Elastic sketch can adapt well to traffic characteristics. Compared to the state-of-the-art, the Elastic sketch achieves 44.6 ∼ 45.2 times faster speed and 2.0 ∼ 273.7 smaller error rate.

[1]  Wei Wang,et al.  Noisy Bloom Filters for Multi-Set Membership Testing , 2016, SIGMETRICS.

[2]  Jin Cao,et al.  Sequential hashing: A flexible approach for unveiling significant patterns in high speed networks , 2010, Comput. Networks.

[3]  Marios Hadjieleftheriou,et al.  Finding frequent items in data streams , 2008, Proc. VLDB Endow..

[4]  Kyu-Young Whang,et al.  A linear-time probabilistic counting algorithm for database applications , 1990, TODS.

[5]  George Varghese,et al.  Hash-Based Techniques for High-Speed Packet Processing , 2010, Algorithms for Next Generation Networks.

[6]  David M. W. Powers,et al.  Applications and Explanations of Zipf’s Law , 1998, CoNLL.

[7]  Bin Fan,et al.  MemC3: Compact and Concurrent MemCache with Dumber Caching and Smarter Hashing , 2013, NSDI.

[8]  George Varghese,et al.  New directions in traffic measurement and accounting: Focusing on the elephants, ignoring the mice , 2003, TOCS.

[9]  Zhi-Li Zhang,et al.  Adaptive packet sampling for accurate and scalable flow measurement , 2004, IEEE Global Telecommunications Conference, 2004. GLOBECOM '04..

[10]  Walter Willinger,et al.  Spatio-temporal compressive sensing and internet traffic matrices , 2009, SIGCOMM '09.

[11]  Jiannong Cao,et al.  On-Line Anomaly Detection With High Accuracy , 2018, IEEE/ACM Transactions on Networking.

[12]  Roy Friedman,et al.  Constant Time Updates in Hierarchical Heavy Hitters , 2017, SIGCOMM.

[13]  Graham Cormode,et al.  An improved data stream summary: the count-min sketch and its applications , 2004, J. Algorithms.

[14]  Graham Cormode,et al.  Sketch Algorithms for Estimating Point Queries in NLP , 2012, EMNLP.

[15]  Andrea Montanari,et al.  Counter braids: a novel counter architecture for per-flow measurement , 2008, SIGMETRICS '08.

[16]  Yan Chen,et al.  Reversible sketches for efficient and accurate change detection over network data streams , 2004, IMC '04.

[17]  David A. Maltz,et al.  Unraveling the Complexity of Network Management , 2009, NSDI.

[18]  Walter Willinger,et al.  On the Self-Similar Nature of Ethernet Traffic ( extended version ) , 1995 .

[19]  Balachander Krishnamurthy,et al.  A manifesto for modeling and measurement in social media , 2010, First Monday.

[20]  Vyas Sekar,et al.  Data streaming algorithms for estimating entropy of network traffic , 2006, SIGMETRICS '06/Performance '06.

[21]  Vladimir Braverman,et al.  One Sketch to Rule Them All: Rethinking Network Flow Monitoring with UnivMon , 2016, SIGCOMM.

[22]  Mario Gerla,et al.  Adaptive forwarding rate control for network coding in tactical manets , 2010, 2010 - MILCOM 2010 MILITARY COMMUNICATIONS CONFERENCE.

[23]  Xiaozhou Li,et al.  Algorithmic improvements for fast concurrent Cuckoo hashing , 2014, EuroSys '14.

[24]  Ming Zhang,et al.  Understanding data center traffic characteristics , 2010, CCRV.

[25]  Theophilus Benson,et al.  Dynamic Prioritization of Traffic in Home Networks , 2015 .

[26]  Partha Kanuparthy,et al.  Performance Characterization of a Commercial Video Streaming Service , 2016, Internet Measurement Conference.

[27]  Hyeontaek Lim,et al.  MICA: A Holistic Approach to Fast In-Memory Key-Value Storage , 2014, NSDI.

[28]  Balachander Krishnamurthy,et al.  Sketch-based change detection: methods, evaluation, and applications , 2003, IMC '03.

[29]  Minyi Guo,et al.  Adaptive Forwarding Delay Control for VANET Data Aggregation , 2012, IEEE Transactions on Parallel and Distributed Systems.

[30]  Haoyu Song,et al.  Fast hash table lookup using extended bloom filter: an aid to network processing , 2005, SIGCOMM '05.

[31]  Ramesh Govindan,et al.  Detection and identification of network anomalies using sketch subspaces , 2006, IMC '06.

[32]  Rafail Ostrovsky,et al.  Generalizing the Layering Method of Indyk and Woodruff: Recursive Sketches for Frequency-Based Vectors on Streams , 2013, APPROX-RANDOM.

[33]  George Varghese,et al.  Carousel: Scalable Logging for Intrusion Prevention Systems , 2010, NSDI.

[34]  Jih-Kwon Peir,et al.  Fit a Spread Estimator in Small Memory , 2009, IEEE INFOCOM 2009.

[35]  Hua Chen,et al.  Pingmesh: A Large-Scale System for Data Center Network Latency Measurement and Analysis , 2015, SIGCOMM.

[36]  Jiannong Cao,et al.  Accurate Recovery of Internet Traffic Data: A Sequential Tensor Completion Approach , 2018, IEEE/ACM Transactions on Networking.

[37]  Burton H. Bloom,et al.  Space/time trade-offs in hash coding with allowable errors , 1970, CACM.

[38]  Alex C. Snoeren,et al.  Inside the Social Network's (Datacenter) Network , 2015, Comput. Commun. Rev..

[39]  Xin Jin,et al.  SketchVisor: Robust Network Measurement for Software Packet Processing , 2017, SIGCOMM.

[40]  Jiannong Cao,et al.  Fast Tensor Factorization for Accurate Internet Anomaly Detection , 2017, IEEE/ACM Transactions on Networking.

[41]  Minlan Yu,et al.  Software Defined Traffic Measurement with OpenSketch , 2013, NSDI.

[42]  Fernando De la Torre,et al.  Facing Imbalanced Data--Recommendations for the Use of Performance Metrics , 2013, 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction.

[43]  Albert G. Greenberg,et al.  The nature of data center traffic: measurements & analysis , 2009, IMC '09.

[44]  Ramesh Govindan,et al.  Evolve or Die: High-Availability Design Principles Drawn from Googles Network Infrastructure , 2016, SIGCOMM.

[45]  Dong Zhou,et al.  Scalable, high performance ethernet forwarding with CuckooSwitch , 2013, CoNEXT.

[46]  Anirudh Sivaraman,et al.  Language-Directed Hardware Design for Network Performance Monitoring , 2017, SIGCOMM.

[47]  Ramesh Govindan,et al.  Trumpet: Timely and Precise Triggers in Data Centers , 2016, SIGCOMM.

[48]  Lili Qiu,et al.  SOAR: Simple Opportunistic Adaptive Routing Protocol for Wireless Mesh Networks , 2009, IEEE Transactions on Mobile Computing.

[49]  Albert G. Greenberg,et al.  Optimizing Cost and Performance in Online Service Provider Networks , 2010, NSDI.

[50]  Ratul Mahajan,et al.  Measuring ISP topologies with Rocketfuel , 2004, IEEE/ACM Transactions on Networking.

[51]  Divyakant Agrawal,et al.  Efficient Computation of Frequent and Top-k Elements in Data Streams , 2005, ICDT.

[52]  Bin Fan,et al.  SILT: a memory-efficient, high-performance key-value store , 2011, SOSP.

[53]  Abhishek Kumar,et al.  Data streaming algorithms for efficient and accurate estimation of flow size distribution , 2004, SIGMETRICS '04/Performance '04.

[54]  Ramesh Govindan,et al.  DREAM , 2014, SIGCOMM.

[55]  Rajeev Motwani,et al.  Approximate Frequency Counts over Data Streams , 2012, VLDB.

[56]  Haipeng Dai,et al.  Finding Persistent Items in Distributed Datasets , 2018, IEEE INFOCOM 2018 - IEEE Conference on Computer Communications.

[57]  Graham Cormode,et al.  Sketch Techniques for Approximate Query Processing , 2010 .

[58]  Minlan Yu,et al.  FlowRadar: A Better NetFlow for Data Centers , 2016, NSDI.

[59]  Moses Charikar,et al.  Finding frequent items in data streams , 2002, Theor. Comput. Sci..

[60]  Haipeng Dai,et al.  Finding Persistent Items in Data Streams , 2016, Proc. VLDB Endow..

[61]  David A. Maltz,et al.  Network traffic characteristics of data centers in the wild , 2010, IMC '10.

[62]  S. Muthukrishnan,et al.  Heavy-Hitter Detection Entirely in the Data Plane , 2016, SOSR.

[63]  David M. W. Powers.  Proceedings of the Joint Conferences on New Methods in Language Processing and Computational Natural Language Learning , 1998 .

[64]  Michael T. Goodrich,et al.  Invertible bloom lookup tables , 2011, 2011 49th Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[65]  Berthold Vöcking,et al.  How asymmetry helps load balancing , 1999, JACM.

[66]  Bin Fan,et al.  Cuckoo Filter: Practically Better Than Bloom , 2014, CoNEXT.