AutoCorrel II: a neural network event correlation approach

As a follow-up to our earlier model Autocorrel I, we have implemented a two-stage event correlation approach with improved performance. Like Autocorrel I, the new model correlates intrusion detection system (IDS) alerts to automate alert and incidents management, and reduce the workload on an IDS analyst. We achieve this correlation by clustering similar alerts, thus allowing the analyst to only consider a few clusters rather than hundreds or thousands of alerts. The first stage uses an artificial neural network (ANN)-based autoassociator (AA). The AA's objective is to attempt to reproduce each alert at its output. In the process, it uses an error metric, the reconstruction error (RE), between its input and output to cluster similar alerts. In order to improve the accuracy of the system we add another machine-learning stage which takes into account the RE as well as raw attribute information from the input alerts. This stage uses the Expectation-Maximisation (EM) clustering algorithm. The performance of this approach is tested with intrusion alerts generated by a Snort IDS on DARPA's 1999 IDS evaluation data as well as incidents.org alerts.

[1]  Jeff A. Bilmes,et al.  A gentle tutorial of the em algorithm and its application to parameter estimation for Gaussian mixture and hidden Markov models , 1998 .

[2]  Teresa F. Lunt,et al.  A survey of intrusion detection techniques , 1993, Comput. Secur..

[3]  Martin Roesch,et al.  Snort - Lightweight Intrusion Detection for Networks , 1999 .

[4]  Anup K. Ghosh,et al.  A Study in Using Neural Networks for Anomaly and Misuse Detection , 1999, USENIX Security Symposium.

[5]  Nathalie Japkowicz,et al.  Supervised Versus Unsupervised Binary-Learning by Feedforward Neural Networks , 2004, Machine Learning.

[6]  R. Lippmann,et al.  An introduction to computing with neural nets , 1987, IEEE ASSP Magazine.

[7]  Yoshua Bengio,et al.  Pattern Recognition and Neural Networks , 1995 .

[8]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[9]  Stephen Northcutt,et al.  Network Intrusion Detection: An Analyst's Hand-book , 1999 .

[10]  Marc Dacier,et al.  Mining intrusion detection alarms for actionable knowledge , 2002, KDD.

[11]  Nathalie Japkowicz,et al.  AutoCorrel: a neural network event correlation approach , 2006, SPIE Defense + Commercial Sensing.

[12]  JapkowiczNathalie.  Supervised Versus Unsupervised Binary-Learning by Feedforward Neural Networks , 2001 .

[13]  Hervé Debar,et al.  Aggregation and Correlation of Intrusion-Detection Alerts , 2001, Recent Advances in Intrusion Detection.

[14]  Bernard Widrow,et al.  Appendix G: Thirty Years of Adaptive Neural Networks: Perceptron, Madaline, and Backpropagation , 2008 .

[15]  Richard Lippmann,et al.  The 1999 DARPA off-line intrusion detection evaluation , 2000, Comput. Networks.

[16]  Hervé Debar,et al.  The Intrusion Detection Message Exchange Format (IDMEF) , 2007, RFC.

[17]  Alfonso Valdes,et al.  Probabilistic Alert Correlation , 2001, Recent Advances in Intrusion Detection.

[18]  Stephen Taylor,et al.  Validation of Sensor Alert Correlators , 2003, IEEE Secur. Priv..

[19]  Bernard Widrow,et al.  30 years of adaptive neural networks: perceptron, Madaline, and backpropagation , 1990, Proc. IEEE.

[20]  Richard P. Lippmann,et al.  An introduction to computing with neural nets , 1987 .

[21]  Robert K. Cunningham,et al.  Fusing A Heterogeneous Alert Stream Into Scenarios , 2002, Applications of Data Mining in Computer Security.

[22]  Nathalie Japkowicz,et al.  A Novelty Detection Approach to Classification , 1995, IJCAI.

[23]  Peng Ning,et al.  An Intrusion Alert Correlator Based on Prerequisites of Intrusions , 2002 .

[24]  Jacek M. Zurada,et al.  Introduction to artificial neural systems , 1992 .

[25]  Isij Monitor,et al.  Network Intrusion Detection: An Analyst’s Handbook , 2000 .

[26]  George M. Mohay,et al.  Attack signature matching and discovery in systems employing heterogeneous IDS , 2003, 19th Annual Computer Security Applications Conference, 2003. Proceedings..