A Three-Stage Process to Detect Outliers and False Positives Generated by Intrusion Detection Systems

To protect computer networks from attacks and hackers, an intrusion detection system (IDS) should be integrated in the security architecture. Although the detection of intrusions and attacks is the ultimate goal, IDSs generate a huge amount of false alerts which cannot be properly managed by the administrator, along with many noisy alerts or outliers. Many research works were conducted to improve IDS's accuracy by reducing the rate of false alerts and eliminating outliers. In this paper, we propose a three-stage process to detect false alerts and outliers. In the first stage, we cluster the set of elementary alerts to create a set of meta-alerts. Then, we remove outliers from the set of meta-alerts using a binary optimization problem. In the last stage, a binary classification algorithm is proposed to classify meta-alerts either as false alerts or real attacks. Experimental results show that our proposed process outperforms concurrent methods by significantly reducing the rate of false alerts and outliers.

[1]  Dimitrios Gunopulos,et al.  Automatic subspace clustering of high dimensional data for data mining applications , 1998, SIGMOD '98.

[2]  Aidong Zhang,et al.  FindOut: Finding Outliers in Very Large Datasets , 2002, Knowledge and Information Systems.

[3]  R. Bone.  Discovery , 1938, Nature.

[4]  Hans-Peter Kriegel,et al.  A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise , 1996, KDD.

[5]  Boleslaw K. Szymanski,et al.  NETWORK-BASED INTRUSION DETECTION USING NEURAL NETWORKS , 2002 .

[6]  Ali A. Ghorbani,et al.  Alert Correlation for Extracting Attack Strategies , 2006, Int. J. Netw. Secur..

[7]  Habiba Drias,et al.  An intrusion detection and alert correlation approach based on revising probabilistic classifiers using expert knowledge , 2012, Applied Intelligence.

[8]  Kenji Yamanishi,et al.  Discovering outlier filtering rules from unlabeled data: combining a supervised learner with an unsupervised learner , 2001, KDD '01.

[9]  Wei-Zhi Wu,et al.  Neighborhood operator systems and approximations , 2002, Inf. Sci..

[10]  Sridhar Ramaswamy,et al.  Efficient algorithms for mining outliers from large data sets , 2000, SIGMOD '00.

[11]  Jian Tang,et al.  Capabilities of outlier detection schemes in large datasets, framework and methodologies , 2006, Knowledge and Information Systems.

[12]  Hongli Zhang,et al.  Intrusion detection alarms reduction using root cause analysis and clustering , 2009, Comput. Commun..

[13]  Maria Papadaki,et al.  The Problem of False Alarms: Evaluation with Snort and DARPA 1999 Dataset , 2008, TrustBus.

[14]  S. S. Dhande.  Outlier Detection over Data Set Using Cluster-Based and Distance-Based Approach , 2012 .

[15]  Tian Zhang,et al.  BIRCH: an efficient data clustering method for very large databases , 1996, SIGMOD '96.

[16]  Ada Wai-Chee Fu,et al.  Enhancements on local outlier detection , 2003, Seventh International Database Engineering and Applications Symposium, 2003. Proceedings..

[17]  Peter J. Rousseeuw,et al.  Robust Regression and Outlier Detection , 2005, Wiley Series in Probability and Statistics.

[18]  Clara Pizzuti,et al.  Fast Outlier Detection in High Dimensional Spaces , 2002, PKDD.

[19]  Shian-Shyong Tseng,et al.  Two-phase clustering process for outliers detection , 2001, Pattern Recognit. Lett..

[20]  Osmar R. Zaïane,et al.  A Nonparametric Outlier Detection for Effectively Discovering Top-N Outliers from Engineering Data , 2006, PAKDD.

[21]  Takafumi Kanamori,et al.  Inlier-Based Outlier Detection via Direct Density Ratio Estimation , 2008, 2008 Eighth IEEE International Conference on Data Mining.

[22]  Javier Lopez,et al.  Trust, Privacy, and Security in Digital Business , 2014, Lecture Notes in Computer Science.

[23]  Chih-Hsuan Wang,et al.  Outlier identification and market segmentation using kernel-based clustering techniques , 2009, Expert Syst. Appl..

[24]  Gregory Piatetsky-Shapiro,et al.  Discovery, Analysis, and Presentation of Strong Rules , 1991, Knowledge Discovery in Databases.

[25]  Srinivasan Parthasarathy,et al.  Fast Distributed Outlier Detection in Mixed-Attribute Data Sets , 2006, Data Mining and Knowledge Discovery.

[26]  Santosh Biswas,et al.  Network specific false alarm reduction in intrusion detection system , 2011, Secur. Commun. Networks.

[27]  Raymond T. Ng,et al.  Algorithms for Mining Distance-Based Outliers in Large Datasets , 1998, VLDB.

[28]  Yan Zhang,et al.  IDS Alert Classification Model Construction Using Decision Support Techniques , 2012, 2012 International Conference on Computer Science and Electronics Engineering.

[29]  Francisco Herrera,et al.  On the combination of genetic fuzzy systems and pairwise learning for improving detection rates on Intrusion Detection Systems , 2015, Expert Syst. Appl..

[30]  Yumin Chen,et al.  Neighborhood outlier detection , 2010, Expert Syst. Appl..

[31]  Anil K. Jain,et al.  Data clustering: a review , 1999, CSUR.

[32]  Bianca Zadrozny,et al.  Outlier detection by active learning , 2006, KDD '06.

[33]  W. R. Buckland,et al.  Outliers in Statistical Data , 1979 .

[34]  Peter J. Rousseeuw,et al.  Robust regression and outlier detection , 1987 .

[35]  Maria Papadaki,et al.  A preliminary two-stage alarm correlation and filtering system using SOM neural network and K-means algorithm , 2010, Comput. Secur..

[36]  Yong Shi,et al.  Towards exploring interactive relationship between clusters and outliers in multi-dimensional data analysis , 2005, 21st International Conference on Data Engineering (ICDE'05).

[37]  Anthony K. H. Tung,et al.  Mining top-n local outliers in large databases , 2001, KDD '01.

[38]  Klaus Julisch,et al.  Clustering intrusion detection alarms to support root cause analysis , 2003, TSEC.

[39]  Chih-Fong Tsai,et al.  CANN: An intrusion detection system based on combining cluster centers and nearest neighbors , 2015, Knowl. Based Syst..

[40]  Huwaida Tagelsir Elshoush,et al.  An Improved Framework for Intrusion Alert Correlation , 2012 .

[41]  Khaled Labib,et al.  NSOM: A Real-Time Network-Based Intrusion Detection System Using Self-Organizing Maps , 2002 .

[42]  Sudipto Guha,et al.  CURE: an efficient clustering algorithm for large databases , 1998, SIGMOD '98.

[43]  Fuling Bian,et al.  Cell-Based Outlier Detection Algorithm: A Fast Outlier Detection Algorithm for Large Datasets , 2008, PAKDD.

[44]  A. K. Bhattacharjee,et al.  IDS alerts classification using knowledge-based evaluation , 2012, 2012 Fourth International Conference on Communication Systems and Networks (COMSNETS 2012).

[45]  Christos Faloutsos,et al.  LOCI: fast outlier detection using the local correlation integral , 2003, Proceedings 19th International Conference on Data Engineering (Cat. No.03CH37405).