A hardware-accelerated infrastructure for flexible sketch-based network traffic monitoring

Sketch-based data streaming algorithms are used in many network traffic monitoring applications to obtain accurate estimates of traffic flow. However, the flexibility is limited as hardware implementation of sketch counters may not be re-used for different measurement tasks. In this paper, we develop a generic hardware infrastructure for collecting flow statistics. The purpose is to achieve the goal of adopting various sketch-based algorithms with arbitrary flow aggregations for monitoring applications and measurement tasks in a flexible manner. Multiple-choice hashing with linear probing scheme is utilized for high-speed counter update process. Simulation results based on real traffic traces for monitoring applications are presented. The proposed hardware infrastructure is implemented on the NetFPGA-10G platform. The system is capable of processing network traffic at 53 Gbps in a worst-case scenario of 64-byte minimum-sized Ethernet frame.

[1]  Nan Hua,et al.  BRICK: a novel exact active statistics counter architecture , 2011, TNET.

[2]  Vyas Sekar,et al.  Revisiting the case for a minimalist approach for network flow monitoring , 2010, IMC '10.

[3]  Mikkel Thorup,et al.  On the k-Independence Required by Linear Probing and Minwise Independence , 2010, TALG.

[4]  Vyas Sekar,et al.  Data streaming algorithms for estimating entropy of network traffic , 2006, SIGMETRICS '06/Performance '06.

[5]  Shigang Chen,et al.  Per-Flow Traffic Measurement Through Randomized Counter Sharing , 2012, IEEE/ACM Trans. Netw..

[6]  Mikkel Thorup,et al.  Tabulation Based 5-Universal Hashing and Linear Probing , 2010, ALENEX.

[7]  Martin May,et al.  Impact of packet sampling on anomaly detection metrics , 2006, IMC '06.

[8]  Theophilus Wellem,et al.  Superspreader detection system on NetFPGA platform , 2014, 2014 ACM/IEEE Symposium on Architectures for Networking and Communications Systems (ANCS).

[9]  Balachander Krishnamurthy,et al.  Sketch-based change detection: methods, evaluation, and applications , 2003, IMC '03.

[10]  Devavrat Shah,et al.  Maintaining Statistics Counters in Router Line Cards , 2002, IEEE Micro.

[11]  Hui Zang,et al.  Is sampled data sufficient for anomaly detection? , 2006, IMC '06.

[12]  George Varghese,et al.  Efficient implementation of a statistics counter architecture , 2003, SIGMETRICS '03.

[13]  Aiko Pras,et al.  Flow Monitoring Explained: From Packet Capture to Data Analysis With NetFlow and IPFIX , 2014, IEEE Communications Surveys & Tutorials.

[14]  Mark Crovella,et al.  Mining anomalies using traffic feature distributions , 2005, SIGCOMM '05.

[15]  Andrea Montanari,et al.  Counter braids: a novel counter architecture for per-flow measurement , 2008, SIGMETRICS '08.

[16]  Theophilus Wellem,et al.  Hardware-assisted estimation of entropy norm for high-speed network traffic , 2014 .

[17]  Anna Pagh,et al.  Linear probing with constant independence , 2006, STOC '07.

[18]  Abhishek Kumar,et al.  Data streaming algorithms for efficient and accurate estimation of flow size distribution , 2004, SIGMETRICS '04/Performance '04.

[19]  Yu Cheng,et al.  Discount Counting for Fast Flow Statistics on Flow Size and Flow Volume , 2014, IEEE/ACM Transactions on Networking.

[20]  Michael Mitzenmacher,et al.  The Power of One Move: Hashing Schemes for Hardware , 2010, IEEE/ACM Trans. Netw..

[21]  Minlan Yu,et al.  Software Defined Traffic Measurement with OpenSketch , 2013, NSDI.

[22]  Graham Cormode,et al.  An improved data stream summary: the count-min sketch and its applications , 2004, J. Algorithms.

[23]  Moses Charikar,et al.  Finding frequent items in data streams , 2004, Theor. Comput. Sci..

[24]  Abhishek Kumar,et al.  Joint data streaming and sampling techniques for detection of super sources and destinations , 2005, IMC '05.

[25]  Danny Wen-Yaw Chung,et al.  A software defined sketch system for traffic monitoring , 2015, 2015 ACM/IEEE Symposium on Architectures for Networking and Communications Systems (ANCS).

[26]  Cristian Estan,et al.  New directions in traffic measurement and accounting , 2001, IMW '01.

[27]  Vladimir Braverman,et al.  Enabling a "RISC" Approach for Software-Defined Monitoring using Universal Streaming , 2015, HotNets.

[28]  Jih-Kwon Peir,et al.  Fit a Compact Spread Estimator in Small High-Speed Memory , 2011, IEEE/ACM Transactions on Networking.

[29]  Qi Zhao,et al.  Design of a novel statistics counter architecture with optimal space and time efficiency , 2006, SIGMETRICS '06/Performance '06.