Volume Anomaly Detection in Data Networks: An Optimal Detection Algorithm vs. the PCA Approach

The crucial future role of Internet in society makes of network monitoring a critical issue for network operators in future network scenarios. The Future Internet will have to cope with new and different anomalies, motivating the development of accurate detection algorithms. This paper presents a novel approach to detect unexpected and large traffic variations in data networks. We introduce an optimal volume anomaly detection algorithm in which the anomaly-free traffic is treated as a nuisance parameter. The algorithm relies on an original parsimonious model for traffic demands which allows detecting anomalies from link traffic measurements, reducing the overhead of data collection. The performance of the method is compared to that obtained with the Principal Components Analysis (PCA) approach. We choose this method as benchmark given its relevance in the anomaly detection literature. Our proposal is validated using data from an operational network, showing how the method outperforms the PCA approach.

[1]  E. Lehmann Testing Statistical Hypotheses , 1960 .

[2]  Albert G. Greenberg,et al.  Fast accurate computation of large-scale IP traffic matrices from link loads , 2003, SIGMETRICS '03.

[3]  Albert G. Greenberg,et al.  Network anomography , 2005, IMC '05.

[4]  Matthew Roughan,et al.  Computation of IP traffic from link , 2003, SIGMETRICS 2003.

[5]  Ramesh Govindan,et al.  Detection and identification of network anomalies using sketch subspaces , 2006, IMC '06.

[6]  Igor V. Nikiforov,et al.  Non-Bayesian Detection and Detectability of Anomalies From a Few Noisy Tomographic Projections , 2007, IEEE Transactions on Signal Processing.

[7]  David Hutchison,et al.  From Detection to Remediation: A Self-Organized System for Addressing Flash Crowd Problems , 2008, 2008 IEEE International Conference on Communications.

[8]  Mikael Johansson,et al.  Traffic matrix estimation on a large IP backbone: a comparison on real data , 2004, IMC '04.

[9]  Calyampudi R. Rao,et al.  Linear Statistical Inference and Its Applications. , 1975 .

[10]  A. Wald Tests of statistical hypotheses concerning several parameters when the number of observations is large , 1943 .

[11]  H. T. Kung,et al.  Use of spectral analysis in defense against DoS attacks , 2002, Global Telecommunications Conference, 2002. GLOBECOM '02. IEEE.

[12]  Jennifer Rexford,et al.  Sensitivity of PCA for traffic anomaly detection , 2007, SIGMETRICS '07.

[13]  Peter W. Glynn,et al.  Internet service performance failure detection , 1998, PERV.

[14]  Paul Barford,et al.  A signal analysis of network traffic anomalies , 2002, IMW '02.

[15]  Mark Crovella,et al.  Mining anomalies using traffic feature distributions , 2005, SIGCOMM '05.

[16]  Chuanyi Ji,et al.  Proactive network fault detection , 1997, Proceedings of INFOCOM '97.

[17]  Carsten Lund,et al.  Estimating point-to-point and point-to-multipoint traffic matrices: an information-theoretic approach , 2005, IEEE/ACM Transactions on Networking.

[18]  Mikael Johansson,et al.  Data-driven traffic engineering: techniques, experiences and challenges , 2006, 2006 3rd International Conference on Broadband Communications, Networks and Systems.

[19]  Christophe Diot,et al.  Diagnosing network-wide traffic anomalies , 2004, SIGCOMM.

[20]  Christophe Diot,et al.  Traffic matrix estimation: existing techniques and new directions , 2002, SIGCOMM 2002.

[21]  Anukool Lakhina,et al.  Multivariate Online Anomaly Detection Using Kernel Recursive Least Squares , 2007, IEEE INFOCOM 2007 - 26th IEEE International Conference on Computer Communications.

[22]  Hongjoong Kim,et al.  A novel approach to detection of intrusions in computer networks via adaptive sequential and batch-sequential change-point detection methods , 2006, IEEE Transactions on Signal Processing.

[23]  G. Nürnberger Approximation by Spline Functions , 1989 .

[24]  S. Vaton,et al.  Robust and Reactive Traffic Engineering for Dynamic Traffic Demands , 2008, 2008 Next Generation Internet Networks.

[25]  Jake D. Brutlag,et al.  Aberrant Behavior Detection in Time Series for Network Monitoring , 2000, LISA.

[26]  Marina Thottan,et al.  Anomaly detection in IP networks , 2003, IEEE Trans. Signal Process..

[27]  Kang G. Shin,et al.  Detecting SYN flooding attacks , 2002, Proceedings.Twenty-First Annual Joint Conference of the IEEE Computer and Communications Societies.

[28]  Donald F. Towsley,et al.  The monitoring and early detection of Internet worms , 2005, IEEE/ACM Transactions on Networking.

[29]  Kavé Salamatian,et al.  Combining filtering and statistical methods for anomaly detection , 2005, IMC '05.

[30]  Mischa Schwartz,et al.  Schemes for fault identification in communication networks , 1995, TNET.

[31]  Robert Nowak,et al.  Internet tomography , 2002, IEEE Signal Process. Mag..

[32]  Calyampudi R. Rao,et al.  Linear statistical inference and its applications , 1965 .

[33]  Balachander Krishnamurthy,et al.  Flash crowds and denial of service attacks: characterization and implications for CDNs and web sites , 2002, WWW.