Detecting denial-of-service attacks with incomplete audit data

With the ever increasing deployment and usage of gigabit networks, traditional network anomaly detection based intrusion detection systems have not scaled accordingly. Most, if not all, systems deployed assume the availability of complete and clean data for the purpose of intrusion detection. We contend that this assumption is not valid. Factors like noise in the audit data, mobility of the nodes and the large amount of network data generated by the network make it difficult to build a normal traffic profile of the network for the purpose of anomaly detection. From this perspective, we present an anomaly detection scheme, called SCAN (stochastic clustering algorithm for network anomaly detection), that has the capability to detect intrusions with high accuracy even when audit data is not complete. We use the expectation-maximization algorithm to cluster the incoming audit data and compute the missing values in the audit data. We improve the speed of convergence of the clustering process by using Bloom filters and data summaries. We evaluate SCAN using the 1999 DARPA/Lincoln Laboratory intrusion detection evaluation dataset.

[1]  H. Javitz,et al.  Detecting Unusual Program Behavior Using the Statistical Component of the Next-generation Intrusion Detection Expert System ( NIDES ) 1 , 1997 .

[2]  Eleazar Eskin,et al.  A GEOMETRIC FRAMEWORK FOR UNSUPERVISED ANOMALY DETECTION: DETECTING INTRUSIONS IN UNLABELED DATA , 2002 .

[3]  Salvatore J. Stolfo,et al.  A Geometric Framework for Unsupervised Anomaly Detection , 2002, Applications of Data Mining in Computer Security.

[4]  H. Toutenburg.  Little, R.J.A. and D.B. Rubin:Statistical analysis with missing data , 1991 .

[5]  Andrei Broder,et al.  Network Applications of Bloom Filters: A Survey , 2004, Internet Math..

[6]  Eleazar Eskin,et al.  Anomaly Detection over Noisy Data using Learned Probability Distributions , 2000, ICML.

[7]  Carlos Ordonez,et al.  FREM: fast and robust EM clustering for large data sets , 2002, CIKM '02.

[8]  Philip K. Chan,et al.  PHAD: packet header anomaly detection for identifying hostile network traffic , 2001 .

[9]  Philip K. Chan,et al.  Learning nonstationary models of normal network traffic for detecting novel attacks , 2002, KDD.

[10]  Raymond T. Ng,et al.  Algorithms for Mining Distance-Based Outliers in Large Datasets , 1998, VLDB.

[11]  Philip K. Chan,et al.  Learning Models of Network Traffic for Detecting Novel Attacks , 2002 .

[12]  Tian Zhang,et al.  BIRCH: an efficient data clustering method for very large databases , 1996, SIGMOD '96.

[13]  U. Fayyad,et al.  Scaling EM (Expectation Maximization) Clustering to Large Databases , 1998 .

[14]  Geoffrey E. Hinton,et al.  A View of the Em Algorithm that Justifies Incremental, Sparse, and other Variants , 1998, Learning in Graphical Models.

[15]  Roderick J. A. Little,et al.  Statistical Analysis with Missing Data , 1988 .

[16]  Sudipto Guha,et al.  ROCK: a robust clustering algorithm for categorical attributes , 1999, Proceedings 15th International Conference on Data Engineering (Cat. No.99CB36337).

[17]  D. Rubin,et al.  Statistical Analysis with Missing Data , 1988 .

[18]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[19]  Burton H. Bloom,et al.  Space/time trade-offs in hash coding with allowable errors , 1970, CACM.

[20]  Sridhar Ramaswamy,et al.  Efficient algorithms for mining outliers from large data sets , 2000, SIGMOD '00.