Online Intrusion Alert Aggregation with Generative Data Stream Modeling

Alert aggregation is an important subtask of intrusion detection. The goal is to identify and to cluster different alerts-produced by low-level intrusion detection systems, firewalls, etc.-belonging to a specific attack instance which has been initiated by an attacker at a certain point in time. Thus, meta-alerts can be generated for the clusters that contain all the relevant information whereas the amount of data (i.e., alerts) can be reduced substantially. Meta-alerts may then be the basis for reporting to security experts or for communication within a distributed intrusion detection system. We propose a novel technique for online alert aggregation which is based on a dynamic, probabilistic model of the current attack situation. Basically, it can be regarded as a data stream version of a maximum likelihood approach for the estimation of the model parameters. With three benchmark data sets, we demonstrate that it is possible to achieve reduction rates of up to 99.96 percent while the number of missing meta-alerts is extremely low. In addition, meta-alerts are generated with a delay of typically only a few seconds after observing the first alert belonging to a new attack instance.

[1]  Michalis Vazirgiannis,et al.  c ○ 2001 Kluwer Academic Publishers. Manufactured in The Netherlands. On Clustering Validation Techniques , 2022 .

[2]  Frédéric Cuppens,et al.  LAMBDA: A Language to Model a Database for Detection of Attacks , 2000, Recent Advances in Intrusion Detection.

[3]  Hermann de Meer,et al.  A Novelty-Driven Approach to Intrusion Alert Correlation Based on Distributed Hash Tables , 2007, 2007 12th IEEE Symposium on Computers and Communications.

[4]  Tadeusz Pietraszek,et al.  Using Adaptive Alert Classification to Reduce False Positives in Intrusion Detection , 2004, RAID.

[5]  Donald W. Bouldin,et al.  A Cluster Separation Measure , 1979, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[6]  Robert K. Cunningham,et al.  Fusing A Heterogeneous Alert Stream Into Scenarios , 2002, Applications of Data Mining in Computer Security.

[7]  Hervé Debar,et al.  Aggregation and Correlation of Intrusion-Detection Alerts , 2001, Recent Advances in Intrusion Detection.

[8]  Philip K. Chan,et al.  An Analysis of the 1999 DARPA/Lincoln Laboratory Evaluation Data for Network Anomaly Detection , 2003, RAID.

[9]  Jie Ma,et al.  Processing Intrusion Detection Alerts in Large-scale Network , 2008, 2008 International Symposium on Electronic Commerce and Security.

[10]  Klaus Julisch,et al.  Using root cause analysis to handle intrusion detection alarms , 2003 .

[11]  Frédéric Cuppens,et al.  Managing alerts in a multi-intrusion detection environment , 2001, Seventeenth Annual Computer Security Applications Conference.

[12]  Alfonso Valdes,et al.  Probabilistic Alert Correlation , 2001, Recent Advances in Intrusion Detection.

[13]  Yakov Rekhter,et al.  Address Allocation for Private Internets , 1994, RFC.

[14]  Tom Fawcett,et al.  Analysis and Visualization of Classifier Performance: Comparison under Imprecise Class and Cost Distributions , 1997, KDD.

[15]  Christopher Krügel,et al.  Comprehensive approach to intrusion detection alert correlation , 2004, IEEE Transactions on Dependable and Secure Computing.

[16]  Stefan Axelsson,et al.  Intrusion Detection Systems: A Survey and Taxonomy , 2002 .

[17]  Jon Postel,et al.  Assigned Numbers , 1979, RFC.

[18]  B. Sick,et al.  A framework for large-scale simulation of collaborative intrusion detection systems , 2008, 2008 IEEE Conference on Soft Computing in Industrial Applications.

[19]  Prabhakar Raghavan,et al.  Computing on data streams , 1999, External Memory Algorithms.

[20]  Nathalie Japkowicz,et al.  Using Unsupervised Learning for Network Alert Correlation , 2008, Canadian Conference on AI.

[21]  Mica R. Endsley,et al.  Theoretical Underpinnings of Situation Awareness, A Critical Review , 2000 .

[22]  F. Autrel,et al.  Using an Intrusion Detection Alert Similarity Operator to Aggregate and Fuse Alerts , 2005 .

[23]  Hiroki Takakura,et al.  A Comprehensive Approach to Detect Unknown Attacks Via Intrusion Detection Alerts , 2007, ASIAN.

[24]  M. Vazirgiannis,et al.  Clustering validity assessment using multi representatives , 2002 .

[25]  J. Dunn.  Well-Separated Clusters and Optimal Fuzzy Partitions , 1974 .

[26]  Martin Roesch,et al.  Snort - Lightweight Intrusion Detection for Networks , 1999 .

[27]  R.K. Cunningham,et al.  Evaluating intrusion detection systems: the 1998 DARPA off-line intrusion detection evaluation , 2000, Proceedings DARPA Information Survivability Conference and Exposition. DISCEX'00.

[28]  Alexander Hofmann,et al.  Fast and Efficient Training of RBF Networks , 2003, ICANN.

[29]  Radford M. Neal.  Pattern Recognition and Machine Learning , 2007, Technometrics.

[30]  Alexander Hofmann.  Alarmaggregation und Interessantheitsbewertung in einem dezentralisierten Angriffserkennungssystem , 2012 .

[31]  Peng Ning,et al.  Techniques and tools for analyzing intrusion alerts , 2004, TSEC.

[32]  Giovanni Vigna,et al.  STATL: An Attack Language for State-Based Intrusion Detection , 2002, J. Comput. Secur..

[33]  John McHugh,et al.  Testing Intrusion detection systems: a critique of the 1998 and 1999 DARPA intrusion detection system evaluations as performed by Lincoln Laboratory , 2000, TSEC.

[34]  Jinoh Kim,et al.  Applying Data Mining Techniques to Analyze Alert Data , 2003, APWeb.