A Modular Traffic Sampling Architecture: Bringing Versatility and Efficiency to Massive Traffic Analysis

The massive traffic volumes and heterogeneity of services in today’s networks urge for flexible, yet simple measurement solutions to assist network management tasks, without impairing network performance. To turn treatable tasks requiring traffic analysis, sampling the traffic has become mandatory, triggering substantial research in the area. Despite that, there is still a lack of an encompassing solution able to support the flexible deployment of sampling techniques in production networks, adequate to diverse traffic scenarios and measurement activities. In this context, this article proposes a modular traffic sampling architecture able to foster the flexible design and deployment of efficient measurement strategies. The architecture is composed of three layers—management plane, control plane and data plane—covering key components to achieve versatile and lightweight measurements in diverse traffic scenarios and measurement activities. Each component of the architecture is described considering the different strategies, technologies and protocols that compose the several stages of a measurement process. Following the proposed architecture, a sampling framework prototype has been developed, providing a fair environment to assess and compare sampling techniques under distinct measurement scenarios, evaluating their performance in balancing computational burden and accuracy. The results have demonstrated the relevance and applicability of the proposed architecture, revealing that a modular and configurable approach to sampling is a step forward for improving sampling scope and efficiency.

[1]  Baek-Young Choi,et al.  Observations on Cisco sampled NetFlow , 2005, PERV.

[2]  Symeon Papavassiliou,et al.  Network anomaly detection and classification via opportunistic sampling , 2009, IEEE Network.

[3]  Jae-Hyun Jun,et al.  DDoS Attack Detection Using Flow Entropy and Packet Sampling on Huge Networks , 2014 .

[4]  Peter Phaal,et al.  Traffic estimation for the largest sou r ces on a n etwork , u sin g pa ck et sampling with limited storage , 2000 .

[5]  Mathias Uslar,et al.  The Common Information Model CIM , 2012 .

[6]  Alan D. George,et al.  Adaptive Sampling for Network Management , 2001, Journal of Network and Systems Management.

[7]  Kun Dai,et al.  Exploiting Adaptive Packet-Sampling Measurements for Multimedia Traffic Classification , 2014 .

[8]  Ravi Sahita,et al.  Structure of Policy Provisioning Information (SPPI) , 2001, RFC.

[9]  Harish Kumar,et al.  Analyzing Statistical Effect of Sampling on Network Traffic Dataset , 2014 .

[10]  Jürgen Schönwälder,et al.  Exporting MIB Variables Using the IP Flow Information Export (IPFIX) Protocol , 2017, RFC.

[11]  Samir Al-Khayatt,et al.  An adaptive statistical sampling technique for computer network traffic , 2010, 2010 7th International Symposium on Communication Systems, Networks & Digital Signal Processing (CSNDSP 2010).

[12]  Pere Barlet-Ros,et al.  Portscan Detection with Sampled NetFlow , 2009, TMA.

[13]  Nick G. Duffield,et al.  Sampling and Filtering Techniques for IP Packet Selection , 2009, RFC.

[14]  Anoop Ghanwani,et al.  Mechanisms for Optimizing Link Aggregation Group (LAG) and Equal-Cost Multipath (ECMP) Component Link Utilization in Networks , 2015, RFC.

[15]  Jeffrey D. Case,et al.  Introduction and Applicability Statements for Internet-Standard Management Framework , 2002, RFC.

[16]  A. Cabellos-Aparicio,et al.  Packet Loss Estimation Using Distributed Adaptive Sampling , 2008, NOMS Workshops 2008 - IEEE Network Operations and Management Symposium Workshops.

[17]  Jorge-Arnulfo Quiané-Ruiz,et al.  Runtime measurements in the cloud , 2010, Proc. VLDB Endow..

[18]  Aiko Pras,et al.  On the Difference between Information Models and Data Models , 2003, RFC.

[19]  P. J. Green,et al.  Density Estimation for Statistics and Data Analysis , 1987 .

[20]  Hui Zang,et al.  Is sampled data sufficient for anomaly detection? , 2006, IMC '06.

[21]  Angela Orebaugh,et al.  Wireshark & Ethereal Network Protocol Analyzer Toolkit (Jay Beale's Open Source Security) , 2006 .

[22]  J. M. Pullen,et al.  Countering denial-of-service attacks using congestion triggered packet sampling and filtering , 2001, Proceedings Tenth International Conference on Computer Communications and Networks (Cat. No.01EX495).

[23]  Nick G. Duffield,et al.  Trajectory sampling for direct traffic observation , 2001, TNET.

[24]  Paulo Carvalho,et al.  Analysing traffic flows through sampling: A comparative study , 2015, 2015 IEEE Symposium on Computers and Communication (ISCC).

[25]  Angela Orebaugh,et al.  Wireshark & Ethereal Network Protocol Analyzer Toolkit , 2007 .

[26]  George Varghese,et al.  New directions in traffic measurement and accounting , 2002, CCRV.

[27]  Nick G. Duffield,et al.  Fair sampling across network flow measurements , 2012, SIGMETRICS '12.

[28]  Srikanth Kandula,et al.  Sampling biases in network path measurements and what to do about it , 2009, IMC '09.

[29]  Aiko Pras,et al.  Flow Monitoring Explained: From Packet Capture to Data Analysis With NetFlow and IPFIX , 2014, IEEE Communications Surveys & Tutorials.

[30]  Georg Carle,et al.  Information Model for Packet Sampling Exports , 2009, RFC.

[31]  Paul Barford,et al.  Improving accuracy in end-to-end packet loss measurement , 2005, SIGCOMM '05.

[32]  Dario Rossi,et al.  Exploiting packet‐sampling measurements for traffic characterization and classification , 2012, Int. J. Netw. Manag..

[33]  Paulo Carvalho,et al.  Computational weight of network traffic sampling techniques , 2014, 2014 IEEE Symposium on Computers and Communications (ISCC).

[34]  Dario Rossi,et al.  On the impact of sampling on traffic monitoring and analysis , 2010, 2010 22nd International Teletraffic Congress (lTC 22).

[35]  Tanja Zseby,et al.  Deployment of Sampling Methods for SLA Validation with Non-Intrusive Measurements , 2002 .

[36]  Lili Yang,et al.  Sampled Based Estimation of Network Traffic Flow Characteristics , 2007, IEEE INFOCOM 2007 - 26th IEEE International Conference on Computer Communications.

[37]  Nick Feamster,et al.  Boosting the scalability of botnet detection using adaptive traffic sampling , 2011, ASIACCS '11.

[38]  Paulo Carvalho,et al.  A multiadaptive sampling technique for cost-effective network measurements , 2013, Comput. Networks.

[39]  Stefano Giordano,et al.  Traffic Sampling Methods for End-to-End QoS Evaluation in Large Heterogeneous Networks , 1998, Comput. Networks.

[40]  Yevgeni Koucheryavy,et al.  Smart Spaces and Next Generation Wired/Wireless Networking , 2010, Lecture Notes in Computer Science.

[41]  Richard Nelson,et al.  Libtrace: a packet capture and analysis library , 2012, CCRV.

[42]  Paulo Carvalho,et al.  A Cooperative Network Monitoring Overlay , 2011, NEW2AN.

[43]  Benoit Claise,et al.  Information Model for IP Flow Information Export (IPFIX) , 2013, RFC.

[44]  Benoit Claise,et al.  Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of IP Traffic Flow Information , 2008, RFC.

[45]  Jürgen Quittek,et al.  Packet Sampling (PSAMP) Protocol Specifications , 2009, RFC.

[46]  Nick G. Duffield,et al.  On Passive One-Way Loss Measurements Using Sampled Flow Statistics , 2009, IEEE INFOCOM 2009.

[47]  Martin May,et al.  Impact of packet sampling on anomaly detection metrics , 2006, IMC '06.

[48]  Jürgen Quittek,et al.  Definitions of Managed Objects for Packet Sampling , 2012, RFC.

[49]  Benoit Claise,et al.  Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of Flow Information , 2013, RFC.

[50]  Philippe Robert,et al.  Deterministic Versus Probabilistic Packet Sampling in the Internet , 2007, ITC.

[51]  Jiankun Hu,et al.  Critical infrastructure protection: Resource efficient sampling to improve detection of less frequent patterns in network traffic , 2010, J. Netw. Comput. Appl..

[52]  Myungjin Lee,et al.  Two Samples are Enough: Opportunistic Flow-level Latency Estimation using NetFlow , 2010, 2010 Proceedings IEEE INFOCOM.

[53]  Mathias Uslar,et al.  The Common Information Model CIM: IEC 61968/61970 and 62325 - A practical introduction to the CIM , 2012 .

[54]  Benoit Claise,et al.  Packet Sampling for Flow Accounting: Challenges and Limitations , 2008, PAM.

[55]  kc claffy,et al.  Application of sampling methodologies to network traffic characterization , 1993, SIGCOMM 1993.

[56]  Bernard W. Silverman,et al.  Density Estimation for Statistics and Data Analysis , 1987 .

[57]  Paulo Carvalho,et al.  Enhancing traffic sampling scope and efficiency , 2013, 2013 IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS).

[58]  Lillian N. Cassel,et al.  Management of sampled real-time network measurements , 1989, [1989] Proceedings. 14th Conference on Local Computer Networks.

[59]  Albert Cabellos-Aparicio,et al.  Analysis of the impact of sampling on NetFlow traffic classification , 2011, Comput. Networks.

[60]  Benoit Claise,et al.  Exporting MIB Variables using the IPFIX Protocol , 1998 .