Beyond Shannon: Characterizing Internet Traffic with Generalized Entropy Metrics

Tracking changes in feature distributions is very important in the domain of network anomaly detection. Unfortunately, these distributions consist of thousands or even millions of data points. This makes tracking, storing and visualizing changes over time a difficult task. A standard technique for capturing and describing distributions in a compact form is the Shannon entropy analysis. Its use for detecting network anomalies has been studied in-depth and several anomaly detection approaches have applied it with considerable success. However, reducing the information about a distribution to a single number deletes important information such as the nature of the change or it might lead to overlooking a large amount of anomalies entirely. In this paper, we show that a generalized form of entropy is better suited to capture changes in traffic features, by exploring different moments. We introduce the Traffic Entropy Spectrum (TES) to analyze changes in traffic feature distributions and demonstrate its ability to characterize the structure of anomalies using traffic traces from a large ISP.

[1]  Bernhard Plattner,et al.  Host behaviour based early detection of worm outbreaks in Internet backbones , 2005, 14th IEEE International Workshops on Enabling Technologies: Infrastructure for Collaborative Enterprise (WETICE'05).

[2]  Jürgen Quittek,et al.  Requirements for IP Flow Information Export (IPFIX) , 2004, RFC.

[3]  G. Wilk,et al.  Tsallis distribution from minimally selected order statistics , 2007 .

[4]  Eddie Kohler,et al.  Observed structure of addresses in IP traffic , 2006, TNET.

[5]  Artur Ziviani,et al.  Network anomaly detection using nonextensive entropy , 2007, IEEE Communications Letters.

[6]  Thierry Dauxois.  Non-Gaussian distributions under scrutiny , 2007 .

[7]  Walter Willinger,et al.  Self-similarity and heavy tails: structural modeling of network traffic , 1998 .

[8]  Grzegorz Wilk,et al.  Example of a possible interpretation of Tsallis entropy , 2007, 0711.3348.

[9]  Ramesh Govindan,et al.  Detection and identification of network anomalies using sketch subspaces , 2006, IMC '06.

[10]  Claude E. Shannon,et al.  Prediction and Entropy of Printed English , 1951 .

[11]  C. Tsallis.  Nonextensive statistics: theoretical, experimental and computational evidences and connections , 1999, cond-mat/9903356.

[12]  C. Tsallis.  Possible generalization of Boltzmann-Gibbs statistics , 1988 .

[13]  Philippe Owezarski,et al.  Non-Gaussian and Long Memory Statistical Characterizations for Internet Traffic with Anomalies , 2007, IEEE Transactions on Dependable and Secure Computing.

[14]  C. Tsallis.  Entropic nonextensivity: a possible measure of complexity , 2000, cond-mat/0010150.

[15]  Christophe Diot,et al.  Diagnosing network-wide traffic anomalies , 2004, SIGCOMM.

[16]  Bernhard Plattner,et al.  Entropy based worm and anomaly detection in fast IP networks , 2005, 14th IEEE International Workshops on Enabling Technologies: Infrastructure for Collaborative Enterprise (WETICE'05).

[17]  Paul Barford,et al.  A signal analysis of network traffic anomalies , 2002, IMW '02.

[18]  Donald F. Towsley,et al.  Detecting anomalies in network traffic using maximum entropy estimation , 2005, IMC '05.