Local outlier factor and stronger one class classifier based hierarchical model for detection of attacks in network intrusion detection dataset

Identification of attacks by a network intrusion detection system (NIDS) is an important task. In signature or rule based detection, the previously encountered attacks are modeled, and signatures/rules are extracted. These rules are used to detect such attacks in future, but in anomaly or outlier detection system, the normal network traffic is modeled. Any deviation from the normal model is deemed to be an outlier/ attack. Data mining and machine learning techniques are widely used in offline NIDS. Unsupervised and supervised learning techniques differ the way NIDS dataset is treated. The characteristic features of unsupervised and supervised learning are finding patterns in data, detecting outliers, and determining a learned function for input features, generalizing the data instances respectively. The intuition is that if these two techniques are combined, better performance may be obtained. Hence, in this paper the advantages of unsupervised and supervised techniques are inherited in the proposed hierarchical model and devised into three stages to detect attacks in NIDS dataset. NIDS dataset is clustered using Dirichlet process (DP) clustering based on the underlying data distribution. Iteratively on each cluster, local denser areas are identified using local outlier factor (LOF) which in turn is discretized into four bins of separation based on LOF score. Further, in each bin the normal data instances are modeled using one class classifier (OCC). A combination of Density Estimation method, Reconstruction method, and Boundary methods are used for OCC model. A product rule combination of the threemethods takes into consideration the strengths of each method in building a stronger OCC model. Any deviation from this model is considered as an attack. Experiments are conducted on KDD CUP’99 and SSENet-2011 datasets. The results show that the proposed model is able to identify attacks with higher detection rate and low false alarms.

[1]  Andrew J. Clark,et al.  Data preprocessing for anomaly based network intrusion detection: A review , 2011, Comput. Secur..

[2]  R. G. M. Helali.  Data Mining Based Network Intrusion Detection System: A Survey , 2008, TeNe.

[3]  Ian H. Witten,et al.  One-Class Classification by Combining Density and Class Probability Estimation , 2008, ECML/PKDD.

[4]  Hans-Peter Kriegel,et al.  LoOP: local outlier probabilities , 2009, CIKM.

[5]  Christopher M. Bishop,et al.  Novelty detection and neural network validation , 1994 .

[6]  Mohamed A. El-Sharkawi,et al.  Elliptical novelty grouping for on-line short-turn detection of excited running rotors , 1999 .

[7]  Vipin Kumar,et al.  Chameleon: Hierarchical Clustering Using Dynamic Modeling , 1999, Computer.

[8]  Amit Sharma.  Cyber Wars: A Paradigm Shift from Means to Ends , 2010 .

[9]  Fabio Roli,et al.  Intrusion detection in computer networks by a modular ensemble of one-class classifiers , 2008, Inf. Fusion.

[10]  Vern Paxson,et al.  Outside the Closed World: On Using Machine Learning for Network Intrusion Detection , 2010, 2010 IEEE Symposium on Security and Privacy.

[11]  Ron Kohavi,et al.  Supervised and Unsupervised Discretization of Continuous Features , 1995, ICML.

[12]  Dorothy E. Denning,et al.  An Intrusion-Detection Model , 1987, IEEE Transactions on Software Engineering.

[13]  Raymond T. Ng,et al.  Algorithms for Mining Distance-Based Outliers in Large Datasets , 1998, VLDB.

[14]  Nizar Bouguila,et al.  Anomaly Intrusion Detection Using Incremental Learning of an Infinite Mixture Model with Feature Selection , 2013, RSKT.

[15]  Sudipto Guha,et al.  CURE: an efficient clustering algorithm for large databases , 1998, SIGMOD '98.

[16]  Phurivit Sangkatsanee,et al.  Practical real-time intrusion detection using machine learning approaches , 2011, Comput. Commun..

[17]  Oleksiy Mazhelis,et al.  One-class classifiers : a review and analysis of suitability in the context of mobile-masquerader detection , 2006, South Afr. Comput. J..

[18]  Vldb Endowment,et al.  The VLDB journal : the international journal on very large data bases. , 1992 .

[19]  Raymond T. Ng,et al.  Distance-based outliers: algorithms and applications , 2000, The VLDB Journal.

[20]  Jaideep Srivastava,et al.  A Comparative Study of Anomaly Detection Schemes in Network Intrusion Detection , 2003, SDM.

[21]  Hui Wang,et al.  A clustering-based method for unsupervised intrusion detections , 2006, Pattern Recognit. Lett..

[22]  Hans-Peter Kriegel,et al.  LOF: identifying density-based local outliers , 2000, SIGMOD '00.

[23]  M. M. Moya,et al.  One-class classifier networks for target recognition applications , 1993 .

[24]  Ester Yen,et al.  Data mining-based intrusion detectors , 2009, Expert Syst. Appl..

[25]  Lih-Chyau Wuu,et al.  Building intrusion pattern miner for Snort network intrusion detection system , 2007, J. Syst. Softw..

[26]  KhanLatifur,et al.  A new intrusion detection system using support vector machines and hierarchical clustering , 2007, VLDB 2007.

[27]  Robert P. W. Duin,et al.  Combining One-Class Classifiers , 2001, Multiple Classifier Systems.

[28]  Andrew H. Sung,et al.  Intrusion detection using neural networks and support vector machines , 2002, Proceedings of the 2002 International Joint Conference on Neural Networks. IJCNN'02 (Cat. No.02CH37290).

[29]  Chris Sanders,et al.  Applied Network Security Monitoring: Collection, Detection, and Analysis , 2013 .

[30]  Raymond T. Ng,et al.  Finding Intensional Knowledge of Distance-Based Outliers , 1999, VLDB.

[31]  S. Selvakumar,et al.  SSENet-2011: A Network Intrusion Detection System dataset and its comparison with KDD CUP 99 dataset , 2011, 2011 Second Asian Himalayas International Conference on Internet (AH-ICI).

[32]  Jiong Yang,et al.  STING: A Statistical Information Grid Approach to Spatial Data Mining , 1997, VLDB.

[33]  Christos Faloutsos,et al.  LOCI: fast outlier detection using the local correlation integral , 2003, Proceedings 19th International Conference on Data Engineering (Cat. No.03CH37405).

[34]  Francisco Herrera,et al.  A Survey of Discretization Techniques: Taxonomy and Empirical Analysis in Supervised Learning , 2013, IEEE Transactions on Knowledge and Data Engineering.

[35]  G. Box,et al.  Bayesian analysis of some outlier problems in time series , 1979 .

[36]  ZhangAidong,et al.  WaveCluster: a wavelet-based clustering approach for spatial data in very large databases , 2000, VLDB 2000.

[37]  Jugal K. Kalita,et al.  Network Anomaly Detection: Methods, Systems and Tools , 2014, IEEE Communications Surveys & Tutorials.

[38]  Andrew H. Sung,et al.  Intrusion Detection Systems Using Adaptive Regression Splines , 2004, ICEIS.

[39]  Arthur Zimek,et al.  Ensembles for unsupervised outlier detection: challenges and research questions a position paper , 2014, SKDD.

[40]  Ming-Yang Su,et al.  A real-time network intrusion detection system for large-scale attacks based on an incremental mining approach , 2009, Comput. Secur..

[41]  Tom Fawcett,et al.  Activity monitoring: noticing interesting changes in behavior , 1999, KDD '99.

[42]  Jiawei Han,et al.  Efficient and Effective Clustering Methods for Spatial Data Mining , 1994, VLDB.

[43]  Bhavani M. Thuraisingham,et al.  A new intrusion detection system using support vector machines and hierarchical clustering , 2007, The VLDB Journal.

[44]  Zoubin Ghahramani,et al.  Unsupervised and Constrained Dirichlet Process Mixture Models for Verb Clustering , 2009 .

[45]  Ali S. Hadi,et al.  Finding Groups in Data: An Introduction to Chster Analysis , 1991 .

[46]  Charu C. Aggarwal,et al.  On Abnormality Detection in Spuriously Populated Data Streams , 2005, SDM.

[47]  Juan E. Tapiador,et al.  Anomaly detection methods in wired networks: a survey and taxonomy , 2004, Comput. Commun..

[48]  Aidong Zhang,et al.  WaveCluster: a wavelet-based clustering approach for spatial data in very large databases , 2000, The VLDB Journal.

[49]  George Karypis,et al.  C HAMELEON : A Hierarchical Clustering Algorithm Using Dynamic Modeling , 1999 .

[50]  Ji Zhang,et al.  Clustering in Dynamic Spatial Databases , 2005, Journal of Intelligent Information Systems.

[51]  Usama M. Fayyad,et al.  Multi-Interval Discretization of Continuous-Valued Attributes for Classification Learning , 1993, IJCAI.

[52]  Anil K. Jain.  Data clustering: 50 years beyond K-means , 2008, Pattern Recognit. Lett..

[53]  Alfonso Valdes,et al.  Next-generation Intrusion Detection Expert System (NIDES)A Summary , 1997 .

[54]  Dit-Yan Yeung,et al.  Parzen-window network intrusion detectors , 2002, Object recognition supported by user interaction for service robots.

[55]  Hesham Altwaijry,et al.  Bayesian based intrusion detection system , 2012, J. King Saud Univ. Comput. Inf. Sci..

[56]  Sara Matzner,et al.  An application of machine learning to network intrusion detection , 1999, Proceedings 15th Annual Computer Security Applications Conference (ACSAC'99).

[57]  Abdolreza Mirzaei,et al.  Intrusion detection using fuzzy association rules , 2009, Appl. Soft Comput..

[58]  Sridhar Ramaswamy,et al.  Efficient algorithms for mining outliers from large data sets , 2000, SIGMOD '00.

[59]  Ji Zhang,et al.  Advancements of Outlier Detection: A Survey , 2013, EAI Endorsed Trans. Scalable Inf. Syst..

[60]  Fabio Roli,et al.  Adversarial attacks against intrusion detection systems: Taxonomy, solutions and open issues , 2013, Inf. Sci..