Data warehousing and data mining techniques for intrusion detection systems

This paper describes data mining and data warehousing techniques that can improve the performance and usability of Intrusion Detection Systems (IDS). Current IDS do not provide support for historical data analysis and data summarization. This paper presents techniques to model network traffic and alerts using a multi-dimensional data model and star schemas. This data model was used to perform network security analysis and detect denial of service attacks. Our data model can also be used to handle heterogeneous data sources (e.g. firewall logs, system calls, net-flow data) and enable up to two orders of magnitude faster query response times for analysts as compared to the current state of the art. We have used our techniques to implement a prototype system that is being successfully used at Army Research Labs. Our system has helped the security analyst in detecting intrusions and in historical data analysis for generating reports on trend analysis.

[1]  Peng Ning,et al.  Learning attack strategies from intrusion alerts , 2003, CCS '03.

[2]  Sushil Jajodia,et al.  ADAM: Detecting Intrusions by Data Mining , 2001 .

[3]  Anoop Singhal,et al.  ANSWER: Network Monitoring Using Object-Oriented Rules , 1998, AAAI/IAAI.

[4]  Sushil Jajodia,et al.  Data Mining for Intrusion Detection , 2005, Data Mining and Knowledge Discovery Handbook.

[5]  Frédéric Cuppens,et al.  Alert correlation in a cooperative intrusion detection framework , 2002, Proceedings 2002 IEEE Symposium on Security and Privacy.

[6]  Petra Perner,et al.  Data Mining - Concepts and Techniques , 2002, Künstliche Intell..

[7]  Ramakrishnan Srikant,et al.  Fast Algorithms for Mining Association Rules in Large Databases , 1994, VLDB.

[8]  Wenke Lee,et al.  Discovering Novel Attack Strategies from INFOSEC Alerts , 2004, ESORICS.

[9]  Leonid Portnoy,et al.  Intrusion detection with unlabeled data using clustering , 2000 .

[10]  Jack Koziol Intrusion Detection with Snort , 2003 .

[11]  Alfonso Valdes,et al.  Adaptive, Model-Based Monitoring for Cyber Attack Detection , 2000, Recent Advances in Intrusion Detection.

[12]  Peter Gluchowski,et al.  Data Warehouse , 1997, Informatik-Spektrum.

[13]  Anoop Singhal,et al.  Design of a data warehouse system for network/web services , 2004, CIKM '04.

[14]  Wenke Lee,et al.  Statistical Causality Analysis of INFOSEC Alert Data , 2003, RAID.

[15]  Salvatore J. Stolfo,et al.  Data Mining Approaches for Intrusion Detection , 1998, USENIX Security Symposium.

[16]  Hervé Debar,et al.  A neural network component for an intrusion detection system , 1992, Proceedings 1992 IEEE Computer Society Symposium on Research in Security and Privacy.

[17]  Tamas Abraham IDDM: Intrusion Detection Using Data Mining Techniques , 2001 .

[18]  Rakesh Agarwal,et al.  Fast Algorithms for Mining Association Rules , 1994, VLDB 1994.

[19]  W. H. Inmon,et al.  Building the data warehouse (2nd ed.) , 1996 .

[20]  Stephen R. Gardner Building the data warehouse , 1998, CACM.

[21]  Salvatore J. Stolfo,et al.  Mining Audit Data to Build Intrusion Detection Models , 1998, KDD.

[22]  Surajit Chaudhuri,et al.  An overview of data warehousing and OLAP technology , 1997, SGMD.

[23]  Peng Ning,et al.  Constructing attack scenarios through correlation of intrusion alerts , 2002, CCS '02.

[24]  Sushil Jajodia,et al.  Detecting Novel Network Intrusions Using Bayes Estimators , 2001, SDM.

[25]  Jaideep Srivastava,et al.  A Comparative Study of Anomaly Detection Schemes in Network Intrusion Detection , 2003, SDM.

[26]  Sushil Jajodia,et al.  Correlating intrusion events and building attack scenarios through attack graph distances , 2004, 20th Annual Computer Security Applications Conference.

[27]  Jennifer Widom,et al.  Database Systems: The Complete Book , 2001 .