Contributions on detection and classification of internet traffic anomalies

Nowadays, one certainty is that traffic is not well behaved, i.e., its pattern is always changing. Several causes have been pointed as responsible for such variations, some of them being extrinsic to the traffic, as the interaction between traffic, be it legitimate or illegitimate. In a practical point of view, traffic irregularities or traffic anomalies can be described as the result of one or more occurrences that change the normal flow of data over a network. Such occurrences can be triggered by different factors, as Denial of Service (DoS) attacks, flash crowds or management operations. Because the occurrence of such misbehaviours can lead to a lack of control over the network (i.e., security, resources or accuracy issues), since a few years the traffic anomaly related domain has become one of the top research areas, with significant and current contributions. For instance, while some work was mainly concerned with the isolation of network failures, other had, as main intention, the statistical prediction of traffic anomalies using mathematical models. While other was concerned with the introduction of new features in order to enhance traffic analysis. Network Anomaly Detection Algorithm - NADA - is an approach that intends to detect, classify and identify traffic anomalies, being a network anomaly an event that is able of introducing some level of variation on measurable network data. Such variations are disturbing since they have potential to deviate network operations from their normal behaviour. The execution of NADA and its accuracy are guaranteed by considering three axis of action: multi-criteria, multi-scale and multi aggregation level. Altogether they allow the detection of traffic anomalies in traffic traces, as well their classification through the definition of traffic profiles, particularly, anomaly traffic profiles. The latter ones are the basis for an anomaly signatures database. Moreover, the use of those three axis allows an anomaly to be detect ed independently of the traffic parameters it affects (multi-criteria axis), its duration (multi-scale axis) and its intensity (multi-level axis). Hence, anomaly detection and classification form a doublet that can be applied at several areas, ranging from network security to traffic engineering or overlay networks, to name a few. Moreover, if IP information of anomalous flows is added to all this knowledge, as NADA do, it will be possible, with minimum effort, to decide the best actions that should be taken in order to control damages from anomaly occurrences - i.e. to have a fully functional detection system.

[1]  Marina Thottan,et al.  Anomaly detection in IP networks , 2003, IEEE Trans. Signal Process..

[2]  Balachander Krishnamurthy,et al.  Sketch-based change detection: methods, evaluation, and applications , 2003, IMC '03.

[3]  George Varghese,et al.  On Scalable Attack Detection in the Network , 2004, IEEE/ACM Transactions on Networking.

[4]  Dorothy E. Denning,et al.  An Intrusion-Detection Model , 1987, IEEE Transactions on Software Engineering.

[5]  Jim Alves-Foss,et al.  NATE: Network Analysis of Anomalous Traffic Events, a low-cost approach , 2001, NSPW '01.

[6]  Mark Crovella,et al.  Mining anomalies using traffic feature distributions , 2005, SIGCOMM '05.

[7]  Philippe Owezarski,et al.  Non-Gaussian and Long Memory Statistical Characterizations for Internet Traffic with Anomalies , 2007, IEEE Transactions on Dependable and Secure Computing.

[8]  Marina Vannucci,et al.  Detecting Traffic Anomalies through Aggregate Analysis of Packet Header Data , 2004, NETWORKING.

[9]  Peter G. Neumann,et al.  Experience with EMERALD to Date , 1999, Workshop on Intrusion Detection and Network Monitoring.

[10]  Hari Balakrishnan,et al.  Fast portscan detection using sequential hypothesis testing , 2004, IEEE Symposium on Security and Privacy, 2004. Proceedings. 2004.

[11]  Walter Willinger,et al.  On the Self-Similar Nature of Ethernet Traffic ( extended version ) , 1995 .

[12]  Jake D. Brutlag,et al.  Aberrant Behavior Detection in Time Series for Network Monitoring , 2000, LISA.

[13]  Kihong Park,et al.  On the relationship between file sizes, transport protocols, and self-similar network traffic , 1996, Proceedings of 1996 International Conference on Network Protocols (ICNP-96).

[14]  Anwar Elwalid,et al.  The Importance of Long-Range Dependence of VBR Video Traffic in ATM Traffic Engineering: Myths and Realities , 1996, SIGCOMM.

[15]  Harold Joseph Highland,et al.  AIN'T misbehaving—A taxonomy of anti-intrusion techniques , 1995 .

[16]  Mark Crovella,et al.  Characterization of network-wide anomalies in traffic flows , 2004, IMC '04.

[17]  Ratul Mahajan,et al.  Controlling High Bandwidth Aggregates in the Network (Extended Version) , 2001 .

[18]  Philippe Owezarski,et al.  TFRC Contribution to Internet QoS Improvement , 2003, QofIS.

[19]  P. Whittle,et al.  Estimation and information in stationary time series , 1953 .

[20]  Paul Barford,et al.  A signal analysis of network traffic anomalies , 2002, IMW '02.

[21]  Stuart E. Schechter,et al.  Fast Detection of Scanning Worm Infections , 2004, RAID.

[22]  Marc Dacier,et al.  Towards a taxonomy of intrusion-detection systems , 1999, Comput. Networks.

[23]  Jin Cao,et al.  Internet Traffic Tends Toward Poisson and Independent as the Load Increases , 2003 .

[24]  Walter Willinger,et al.  Experimental queueing analysis with long-range dependent packet traffic , 1996, TNET.

[25]  A. L. Narasimha Reddy,et al.  A study of analyzing network traffic as images in real-time , 2005, Proceedings IEEE 24th Annual Joint Conference of the IEEE Computer and Communications Societies..

[26]  Gürsel Serpen,et al.  Why machine learning algorithms fail in misuse detection on KDD intrusion detection data set , 2004, Intell. Data Anal..

[27]  James Won-Ki Hong,et al.  A flow-based method for abnormal network traffic detection , 2004, 2004 IEEE/IFIP Network Operations and Management Symposium (IEEE Cat. No.04CH37507).

[28]  George Varghese,et al.  Automatically inferring patterns of resource consumption in network traffic , 2003, SIGCOMM '03.

[29]  Hyun Joo Kim,et al.  Network Traffic Anomaly Detection based on Ratio and Volume Analysis , 2006 .

[30]  Michalis Faloutsos,et al.  A nonstationary Poisson view of Internet traffic , 2004, IEEE INFOCOM 2004.

[31]  Magnus Almgren,et al.  Consolidation and Evaluation of IDS Taxonomies , 2003 .

[32]  Brian Trammell An IPFIX-Based File Format , 2007 .

[33]  Kathleen A. Jackson INTRUSION DETECTION SYSTEM (IDS) PRODUCT SURVEY , 1999 .

[34]  Kensuke Fukuda,et al.  Extracting hidden anomalies using sketch and non Gaussian multiresolution statistical detection procedures , 2007, LSAD '07.

[35]  H. E. Hurst,et al.  Long-Term Storage Capacity of Reservoirs , 1951 .

[36]  J. R. Wallis,et al.  Computer Experiments With Fractional Gaussian Noises: Part 1, Averages and Variances , 1969 .

[37]  Vincent Kanade,et al.  Clustering Algorithms , 2021, Wireless RF Energy Transfer in the Massive IoT Era.

[38]  Larry Niven,et al.  The Flight of the Horse , 1973 .

[39]  Martin Roesch,et al.  Snort - Lightweight Intrusion Detection for Networks , 1999 .

[40]  J. Chow An Assessment of the DARPA IDS Evaluation Dataset Using Snort S Terry Brugger , 2005 .

[41]  Walter Willinger,et al.  Self-Similar Network Traffic and Performance Evaluation , 2000 .

[42]  Patrice Abry,et al.  Sketch based Anomaly Detection, Identification and Performance Evaluation , 2007 .

[43]  Murad S. Taqqu,et al.  Theory and applications of long-range dependence , 2003 .

[44]  Balachander Krishnamurthy,et al.  Flash crowds and denial of service attacks: characterization and implications for CDNs and web sites , 2002, WWW.

[45]  Matthias Grossglauser,et al.  On the relevance of long-range dependence in network traffic , 1996, SIGCOMM '96.

[46]  George C. Polyzos,et al.  A Parameterizable Methodology for Internet Traffic Flow Profiling , 1995, IEEE J. Sel. Areas Commun..

[47]  Philip K. Chan,et al.  An Analysis of the 1999 DARPA/Lincoln Laboratory Evaluation Data for Network Anomaly Detection , 2003, RAID.

[48]  Konstantina Papagiannaki,et al.  Network performance monitoring at small time scales , 2003, IMC '03.

[49]  Karl N. Levitt,et al.  Execution monitoring of security-critical programs in distributed systems: a specification-based approach , 1997, Proceedings. 1997 IEEE Symposium on Security and Privacy (Cat. No.97CB36097).

[50]  Mark Crovella,et al.  Diagnosing network-wide traffic anomalies , 2004, SIGCOMM '04.

[51]  R.K. Cunningham,et al.  Evaluating intrusion detection systems: the 1998 DARPA off-line intrusion detection evaluation , 2000, Proceedings DARPA Information Survivability Conference and Exposition. DISCEX'00.

[52]  Azer Bestavros,et al.  Self-similarity in World Wide Web traffic: evidence and possible causes , 1996, SIGMETRICS '96.

[53]  Jan Beran,et al.  Statistics for long-memory processes , 1994 .

[54]  J. MacQueen Some methods for classification and analysis of multivariate observations , 1967 .

[55]  Carsten Lund,et al.  Online identification of hierarchical heavy hitters: algorithms, evaluation, and applications , 2004, IMC '04.

[56]  B. Karp,et al.  Autograph: Toward Automated, Distributed Worm Signature Detection , 2004, USENIX Security Symposium.

[57]  Don H. Johnson,et al.  the Kullback-Leibler distance , 2001 .

[58]  Walter Willinger,et al.  Self-similarity through high-variability: statistical analysis of Ethernet LAN traffic at the source level , 1997, TNET.

[59]  S. Joe Qin,et al.  Subspace approach to multidimensional fault identification and reconstruction , 1998 .

[60]  Eddie Kohler,et al.  Observed Structure of Addresses in IP Traffic , 2002, IEEE/ACM Transactions on Networking.

[61]  John S. Heidemann,et al.  A framework for classifying denial of service attacks , 2003, SIGCOMM '03.

[62]  Vyas Sekar,et al.  Analyzing large DDoS attacks using multiple data sources , 2006, LSAD '06.

[63]  Philip K. Chan,et al.  PHAD: packet header anomaly detection for identifying hostile network traffic , 2001 .

[64]  Mark E. Crovella,et al.  Effect of traffic self-similarity on network performance , 1997, Other Conferences.

[65]  Graham Cormode,et al.  What's new: finding significant differences in network data streams , 2004, IEEE/ACM Transactions on Networking.

[66]  Philippe Owezarski On the impact of DoS attacks on Internet traffic characteristics and QoS , 2005, Proceedings. 14th International Conference on Computer Communications and Networks, 2005. ICCCN 2005..

[67]  John McHugh,et al.  Testing Intrusion detection systems: a critique of the 1998 and 1999 DARPA intrusion detection system evaluations as performed by Lincoln Laboratory , 2000, TSEC.

[68]  Elena Baralis,et al.  Data mining techniques for effective and scalable traffic analysis , 2005, 2005 9th IFIP/IEEE International Symposium on Integrated Network Management, 2005. IM 2005..

[69]  Benoit B. Mandelbrot,et al.  Fractal Geometry of Nature , 1984 .

[70]  Dominique Alessandri,et al.  Towards a Taxonomy of Intrusion Detection Systems and Attacks , 2001 .

[71]  Will E. Leland,et al.  High time-resolution measurement and analysis of LAN traffic: Implications for LAN interconnection , 1991, IEEE INFCOM '91. The conference on Computer Communications. Tenth Annual Joint Comference of the IEEE Computer and Communications Societies Proceedings.

[72]  Philippe Owezarski,et al.  Internet Traffic Characterization - An Analysis of Traffic Oscillations , 2004, HSNMC.

[73]  Salvatore J. Stolfo,et al.  Mining in a data-flow environment: experience in network intrusion detection , 1999, KDD '99.