goProbe: a scalable distributed network monitoring solution

The Internet has developed into the primary means of communication, while ensuring availability and stability is becoming an increasingly challenging task. Traffic monitoring enables network operators to comprehend the composition of traffic flowing through individual corporate and private networks, making it essential for planning, reporting and debugging purposes. Classical packet capture and aggregation concepts (e.g. NetFlow) typically rely on centralized collection of traffic metadata. With the proliferation of network enabled devices and the resulting increase in data volume, such approaches suffer from scalability issues, often prohibiting the transfer of raw metadata as such. This paper describes a decentralized approach, eliminating the need for a central collector and storing local views of network traffic patterns on the respective devices performing the capture. In order to allow for the analysis of captured data, queries formulated by analysts are distributed across all devices. Processing takes place in a parallelized fashion on the respective local data. Consequently, instead of continually transferring raw metadata, significantly smaller aggregate results are sent to a central location which are then combined into the requested final result. The proposed system describes a lightweight and scalable monitoring solution, enabling the efficient use of available system resources on the distributed devices, hence allowing for high performance, real-time traffic analysis on a global scale. The solution was implemented and deployed globally on hosts managed and maintained by a large managed network security services provider.

[1]  Fernando Gont,et al.  Recommendations for Transport-Protocol Port Randomization , 2011, RFC.

[2]  Burkhard Stiller,et al.  DIPStorage: Distributed storage of IP flow records , 2008, 2008 16th IEEE Workshop on Local and Metropolitan Area Networks.

[3]  Andrey Gubarev,et al.  Dremel: Interactive Analysis of Web-Scale Datasets , 2010, Proc. VLDB Endow..

[4]  Alexander Zeier,et al.  Speeding Up Queries in Column Stores - A Case for Compression , 2010, DaWak.

[5]  Yin Zhang,et al.  STAR: Self-Tuning Aggregation for Scalable Monitoring , 2007, VLDB.

[6]  Antonio Manuel Pina,et al.  Two High-Performance Alternatives to ZLIB Scientific-Data Compression , 2014, ICCSA.

[7]  Luca Deri nProbe: an Open Source NetFlow Probe for Gigabit Networks , 2003 .

[8]  Prabhat,et al.  FastBit: interactively searching massive data , 2009 .

[9]  Graham Cormode,et al.  What's hot and what's not: tracking most frequent items dynamically , 2003, PODS.

[10]  Michael J. Freedman,et al.  Aggregation and Degradation in JetStream: Streaming Analytics in the Wide Area , 2014, NSDI.

[11]  Brian Trammell,et al.  YAF: Yet Another Flowmeter , 2010, LISA.

[12]  Dominik Slezak,et al.  Data warehouse technology by infobright , 2009, SIGMOD Conference.

[13]  Steven McCanne,et al.  The BSD Packet Filter: A New Architecture for User-level Packet Capture , 1993, USENIX Winter.

[14]  Daniel J. Abadi,et al.  Column-stores vs. row-stores: how different are they really? , 2008, SIGMOD Conference.

[15]  Daniel Keren,et al.  Sketch-based Geometric Monitoring of Distributed Stream Queries , 2013, Proc. VLDB Endow..

[16]  Robbert van Renesse,et al.  Astrolabe: A robust and scalable technology for distributed system monitoring, management, and data mining , 2003, TOCS.

[17]  Stuart Cheshire,et al.  Internet Assigned Numbers Authority (IANA) Procedures for the Management of the Service Name and Transport Protocol Port Number Registry , 2011, RFC.

[18]  James Won-Ki Hong,et al.  The Architecture of NG-MON: A Passive Network Monitoring System for High-Speed IP Networks , 2002, DSOM.

[19]  Marco Danelutto,et al.  Deep Packet Inspection on Commodity Hardware using FastFlow , 2013, PARCO.

[20]  Yanbo Han,et al.  Engineering and Deployment of Cooperative Information Systems , 2002, Lecture Notes in Computer Science.

[21]  Benoit Claise,et al.  Cisco Systems NetFlow Services Export Version 9 , 2004, RFC.

[22]  Graham Cormode,et al.  Communication-efficient distributed monitoring of thresholded counts , 2006, SIGMOD Conference.

[23]  Frederic P. Miller,et al.  Advanced Encryption Standard , 2009 .

[24]  Tomasz Bujlow Classification and Analysis of Computer Network Traffic , 2014 .