DATALITE: a distributed architecture for traffic analysis via light-weight traffic digest

In this paper, we propose DATALITE, a Distributed Architecture for Traffic Analysis via LIght-weight Traffic digEst, which introduces a set of new distributed algorithms and protocols to support general Traffic Measurement and Analysis (TMA) functions for large-scale, 10Gbps+ packet-switched networks. We formulate the network-wide traffic measurement/ analysis problem as a series of set-cardinality-determination (SCD) problems. By leveraging recent advances in probabilistic distinct sample counting techniques, the set-cardinalities, and thus, the network-wide traffic measurements of interest can be computed in a distributed manner via the exchange of extremely light-weight traffic digests (TD’s) amongst the network nodes. A TD for N packets only requires O(loglog N) bits of memory storage.

[1]  Srikanta Tirthapura,et al.  Estimating simple functions on the union of data streams , 2001, SPAA '01.

[2]  Carsten Lund,et al.  Estimating point-to-point and point-to-multipoint traffic matrices: an information-theoretic approach , 2005, IEEE/ACM Transactions on Networking.

[3]  Kavé Salamatian,et al.  Traffic matrix estimation: existing techniques and new directions , 2002, SIGCOMM '02.

[4]  Bing Yu,et al.  Time-Varying Network Tomography: Router Link Data , 2000 .

[5]  Albert G. Greenberg,et al.  Fast accurate computation of large-scale IP traffic matrices from link loads , 2003, SIGMETRICS '03.

[6]  Y. Vardi,et al.  Network Tomography: Estimating Source-Destination Traffic Intensities from Link Data , 1996 .

[7]  Andrei Broder,et al.  Network Applications of Bloom Filters: A Survey , 2004, Internet Math..

[8]  Philippe Flajolet,et al.  Probabilistic Counting Algorithms for Data Base Applications , 1985, J. Comput. Syst. Sci..

[9]  Jianping Pan,et al.  Fast and accurate traffic matrix measurement using adaptive cardinality counting , 2005, MineNet '05.

[10]  Andrei Z. Broder,et al.  On the resemblance and containment of documents , 1997, Proceedings. Compression and Complexity of SEQUENCES 1997 (Cat. No.97TB100171).

[11]  Kai Hwang,et al.  A Scalable Set-Union Counting Approach to Pushing Back DDoS Attacks , 2004 .

[12]  George Varghese,et al.  Counting the number of active flows on a high speed link , 2002, CCRV.

[13]  Anna R. Karlin,et al.  Network support for IP traceback , 2001, TNET.

[14]  Alan M. Frieze,et al.  Min-Wise Independent Permutations , 2000, J. Comput. Syst. Sci..

[15]  Craig Partridge,et al.  Hash-based IP traceback , 2001, SIGCOMM.

[16]  Nick G. Duffield,et al.  Trajectory sampling for direct traffic observation , 2001, TNET.

[17]  Jennifer Widom,et al.  Models and issues in data stream systems , 2002, PODS.

[18]  Alex C. Snoeren,et al.  Hash-based IP traceback , 2001, SIGCOMM '01.

[19]  Steven M. Bellovin,et al.  Implementing Pushback: Router-Based Defense Against DDoS Attacks , 2002, NDSS.

[20]  Burton H. Bloom,et al.  Space/time trade-offs in hash coding with allowable errors , 1970, CACM.

[21]  Jeffrey Considine,et al.  Informed content delivery across adaptive overlay networks , 2002, IEEE/ACM Transactions on Networking.

[22]  P. Flajolet,et al.  Loglog counting of large cardinalities , 2003 .

[23]  Carsten Lund,et al.  An information-theoretic approach to traffic matrix estimation , 2003, SIGCOMM '03.