Fast and accurate traffic matrix measurement using adaptive cardinality counting

Traffic matrix (TM) can be used to detect, identify, and trace network anomaly caused by DDoS attacks and worm outbreaks. To detect network anomaly as early as possible, we need to obtain TM in a fast and accurate manner. Many existing TM estimation techniques are found not sufficient for this purpose due to their high overhead or low accuracy. We propose a cardinality-based TM measurement approach with an adaptive counting algorithm to produce both packetlevel and flow-level TM, which is well-suited for TM-based anomaly detection on a network basis. Our results show that the approach can obtain TM in almost real-time (once very 10 seconds) with low average relative error (less than 5%). Our approach has low processing, storage and communication overhead, e.g. software implementation can support OC-192 line speed. It can also be implemented in a passive mode and deployed incrementally without changing current routing infrastructure.

[1]  Christophe Diot,et al.  Diagnosing network-wide traffic anomalies , 2004, SIGCOMM.

[2]  Christophe Diot,et al.  Taxonomy of IP traffic matrices , 2002, SPIE ITCom.

[3]  Abhishek Kumar,et al.  Data streaming algorithms for efficient and accurate estimation of flow size distribution , 2004, SIGMETRICS '04/Performance '04.

[4]  An 80 Gbps FPGA Implementation of a Universal Hash Function based Message Authentication Code , 2004 .

[5]  Y. Vardi,et al.  Network Tomography: Estimating Source-Destination Traffic Intensities from Link Data , 1996 .

[6]  Philippe Flajolet,et al.  Probabilistic Counting Algorithms for Data Base Applications , 1985, J. Comput. Syst. Sci..

[7]  Christophe Diot,et al.  Pop-level and access-link-level traffic dynamics in a tier-1 POP , 2001, IMW '01.

[8]  Hugo Krawczyk,et al.  UMAC: Fast and Secure Message Authentication , 1999, CRYPTO.

[9]  Albert G. Greenberg,et al.  Fast accurate computation of large-scale IP traffic matrices from link loads , 2003, SIGMETRICS '03.

[10]  Craig Partridge,et al.  Single-packet IP traceback , 2002, TNET.

[11]  George Varghese,et al.  Bitmap algorithms for counting active flows on high speed links , 2003, IMC '03.

[12]  Konstantina Papagiannaki,et al.  A distributed approach to measure IP traffic matrices , 2004, IMC '04.

[13]  Emilio Leonardi,et al.  How to identify and estimate the largest traffic matrix elements in a dynamic environment , 2004, SIGMETRICS '04/Performance '04.

[14]  Carsten Lund,et al.  Estimating flow distributions from sampled flow statistics , 2003, SIGCOMM '03.

[15]  Michael A. West,et al.  Bayesian Inference on Network Traffic Using Link Count Data , 1998 .

[16]  David L. Mills,et al.  Improved algorithms for synchronizing computer network clocks , 1995, TNET.

[17]  Carsten Lund,et al.  An information-theoretic approach to traffic matrix estimation , 2003, SIGCOMM '03.

[18]  Kyu-Young Whang,et al.  A linear-time probabilistic counting algorithm for database applications , 1990, TODS.

[19]  B. Yu,et al.  Time-varying network tomography: router link data , 2000, 2000 IEEE International Symposium on Information Theory (Cat. No.00CH37060).

[20]  P. Flajolet,et al.  Loglog counting of large cardinalities , 2003 .