FastFlow: A Framework for Accurate Characterization of Network Traffic

This paper proposes a new measurement architecture and associated traffic estimation algorithm called FastFlow that uses the heavy-tailed nature of Internet traffic in order to distinguish packets belonging to short lived flows (SLFs) and long lived flows (LLFs). While complete information is stored for SLFs, only partial information related to LLFs is collected using systematic sampling. The absence of data points in LLFs is approximated using a likelihood function defined over the coupon collector problem and the distribution of underlying traffic estimated using the non-parametric Parzen window technique. We validate the performance of our approach using traffic traces collected from our lab and observe that the estimated statistics match the observed traces with high accuracy.

[1]  Jin Cao,et al.  Internet Traffic Tends Toward Poisson and Independent as the Load Increases , 2003 .

[2]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[3]  E. Parzen.  On Estimation of a Probability Density Function and Mode , 1962 .

[4]  Rajeev Gandhi,et al.  Sluice: Secure Dissemination of Code Updates in Sensor Networks , 2005, 26th IEEE International Conference on Distributed Computing Systems (ICDCS'06).

[5]  Carsten Lund,et al.  Properties and prediction of flow statistics from sampled packet streams , 2002, IMW '02.

[6]  Devavrat Shah,et al.  Maintaining Statistics Counters in Router Line Cards , 2002, IEEE Micro.

[7]  Jie Liu,et al.  Greedy is Good: On Service Tree Placement for In-Network Stream Processing , 2006, 26th IEEE International Conference on Distributed Computing Systems (ICDCS'06).

[8]  Sariinas Ra Ud Ys.  ON THE EFFECTIVENESS OF PARZEN WINDOW CLASSIFIER , 1991 .

[9]  kc claffy,et al.  Understanding Internet traffic streams: dragonflies and tortoises , 2002, IEEE Commun. Mag..

[10]  George Varghese,et al.  Efficient implementation of a statistics counter architecture , 2003, SIGMETRICS '03.

[11]  P. J. Green,et al.  Density Estimation for Statistics and Data Analysis , 1987 .

[12]  Roger Wattenhofer,et al.  Fault-Tolerant Clustering in Ad Hoc and Sensor Networks , 2006, 26th IEEE International Conference on Distributed Computing Systems (ICDCS'06).

[13]  A. Kumar,et al.  Space-code bloom filter for efficient per-flow traffic measurement , 2004, IEEE INFOCOM 2004.

[14]  Carsten Lund,et al.  Flow sampling under hard resource constraints , 2004, SIGMETRICS '04/Performance '04.

[15]  Abhishek Kumar,et al.  Data streaming algorithms for efficient and accurate estimation of flow size distribution , 2004, SIGMETRICS '04/Performance '04.

[16]  Xiaodong Zhang,et al.  ASAP: an AS-Aware Peer-Relay Protocol for High Quality VoIP , 2006, 26th IEEE International Conference on Distributed Computing Systems (ICDCS'06).

[17]  George Varghese,et al.  Bitmap algorithms for counting active flows on high speed links , 2003, IMC '03.

[18]  Eddie Kohler,et al.  The Click modular router , 1999, SOSP.

[19]  Nicola Santoro,et al.  Cycling Through a Dangerous Network: A Simple Efficient Strategy for Black Hole Search , 2006, 26th IEEE International Conference on Distributed Computing Systems (ICDCS'06).

[20]  Guido Appenzeller,et al.  Sizing router buffers , 2004, SIGCOMM '04.

[21]  Nick McKeown,et al.  Monitoring very high speed links , 2001, IMW '01.

[22]  Howard G. Tucker,et al.  Confidence intervals for the number of unseen types , 1998 .

[23]  George Varghese,et al.  New directions in traffic measurement and accounting: Focusing on the elephants, ignoring the mice , 2003, TOCS.

[24]  Nicolas Hohn,et al.  Inverting sampled traffic , 2003, IEEE/ACM Transactions on Networking.

[25]  Carsten Lund,et al.  Estimating flow distributions from sampled flow statistics , 2005, TNET.

[26]  J. A. Buzacott,et al.  On the approximations to the single server queue , 1980 .

[27]  Nathan L. Binkert,et al.  Network-Oriented Full-System Simulation using M5 , 2003 .

[28]  Navendu Jain,et al.  Adaptive Control of Extreme-scale Stream Processing Systems , 2006, 26th IEEE International Conference on Distributed Computing Systems (ICDCS'06).