Anomaly Detection Framework Using Rule Extraction for Efficient Intrusion Detection

Huge datasets in cyber security, such as network trac logs, can be analyzed using machine learning and data mining methods. However, the amount of collected data is increasing, which makes analysis more dicult. Many machine learning methods have not been designed for big datasets, and consequently are slow and dicult to understand. We address the issue of ecient network trac classication by creating an intrusion detection framework that applies dimensionality reduction and conjunctive rule extraction. The system can perform unsupervised anomaly detection and use this information to create conjunctive rules that classify huge amounts of trac in real time. We test the implemented system with the widely used KDD Cup 99 dataset and real-world network logs to conrm that the performance is satisfactory. This system is transparent and does not work like a black box, making it intuitive for domain experts, such as network administrators.

[1]  Stéphane Lafon,et al.  Diffusion maps , 2006 .

[2]  Gabriel Maciá-Fernández,et al.  Anomaly-based network intrusion detection: Techniques, systems and challenges , 2009, Comput. Secur..

[3]  Bart Baesens,et al.  Recursive Neural Network Rule Extraction for Data With Mixed Attributes , 2008, IEEE Transactions on Neural Networks.

[4]  Ian T. Jolliffe,et al.  Principal Component Analysis , 2002, International Encyclopedia of Statistical Science.

[5]  P. Rousseeuw.  Silhouettes: a graphical aid to the interpretation and validation of cluster analysis , 1987 .

[6]  Ulrike von Luxburg,et al.  A tutorial on spectral clustering , 2007, Stat. Comput..

[7]  James Newsome,et al.  Paragraph: Thwarting Signature Learning by Training Maliciously , 2006, RAID.

[8]  Joachim Diederich,et al.  Learning-Based Rule-Extraction From Support Vector Machines: Performance On Benchmark Data Sets , 2004 .

[9]  Pierre Baldi,et al.  Assessing the accuracy of prediction algorithms for classification: an overview , 2000, Bioinform..

[10]  Dawn Xiaodong Song,et al.  Limits of Learning-based Signature Generation with Adversaries , 2008, NDSS.

[11]  Sotiris Ioannidis,et al.  Gnort: High Performance Network Intrusion Detection Using Graphics Processors , 2008, RAID.

[12]  Georgios Kambourakis,et al.  Swarm intelligence in intrusion detection: A survey , 2011, Comput. Secur..

[13]  Anil K. Jain.  Data clustering: 50 years beyond K-means , 2008, Pattern Recognit. Lett..

[14]  Tuomo Sipola,et al.  Combining conjunctive rule extraction with diffusion maps for network intrusion detection , 2013, 2013 IEEE Symposium on Computers and Communications (ISCC).

[15]  Qing He,et al.  Parallel K-Means Clustering Based on MapReduce , 2009, CloudCom.

[16]  John McHugh,et al.  Testing Intrusion detection systems: a critique of the 1998 and 1999 DARPA intrusion detection system evaluations as performed by Lincoln Laboratory , 2000, TSEC.

[17]  Tobias Scheffer,et al.  Learning to Identify Regular Expressions that Describe Email Campaigns , 2012, ICML.

[18]  Kunle Olukotun,et al.  Map-Reduce for Machine Learning on Multicore , 2006, NIPS.

[19]  Jude W. Shavlik,et al.  Using Sampling and Queries to Extract Rules from Trained Neural Networks , 1994, ICML.

[20]  Michael I. Jordan,et al.  Learning Spectral Clustering, With Application To Speech Separation , 2006, J. Mach. Learn. Res..

[21]  Andrew J. Clark,et al.  Data preprocessing for anomaly based network intrusion detection: A review , 2011, Comput. Secur..

[22]  Tuomo Sipola,et al.  Anomaly Detection from Network Logs Using Diffusion Maps , 2011, EANN/AIAI.

[23]  Julie Greensmith,et al.  Immune system approaches to intrusion detection – a review , 2004, Natural Computing.

[24]  Philip K. Chan,et al.  An Analysis of the 1999 DARPA/Lincoln Laboratory Evaluation Data for Network Anomaly Detection , 2003, RAID.

[25]  Jennifer Rexford,et al.  Sensitivity of PCA for traffic anomaly detection , 2007, SIGMETRICS '07.

[26]  Christopher D. Brown,et al.  Receiver operating characteristics curves and related decision measures: A tutorial , 2006 .

[27]  Ali A. Ghorbani,et al.  A detailed analysis of the KDD CUP 99 data set , 2009, 2009 IEEE Symposium on Computational Intelligence for Security and Defense Applications.

[28]  Vern Paxson,et al.  Outside the Closed World: On Using Machine Learning for Network Intrusion Detection , 2010, 2010 IEEE Symposium on Security and Privacy.

[29]  Joachim Diederich,et al.  Rule Extraction from Support Vector Machines , 2008, Studies in Computational Intelligence.

[30]  Gürsel Serpen,et al.  Why machine learning algorithms fail in misuse detection on KDD intrusion detection data set , 2004, Intell. Data Anal..

[31]  Tuomo Sipola,et al.  Adaptive framework for network traffic classification using dimensionality reduction and clustering , 2012, 2012 IV International Congress on Ultra Modern Telecommunications and Control Systems.

[32]  Pere Barlet-Ros,et al.  Operational experiences with anomaly detection in backbone networks , 2012, Comput. Secur..

[33]  Chun-Hung Richard Lin,et al.  Intrusion detection system: A comprehensive review , 2013, J. Netw. Comput. Appl..

[34]  James Newsome,et al.  Polygraph: automatically generating signatures for polymorphic worms , 2005, 2005 IEEE Symposium on Security and Privacy (S&P'05).

[35]  Ann B. Lee,et al.  Geometric diffusions as a tool for harmonic analysis and structure definition of data: diffusion maps. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[36]  John Langford,et al.  Scaling up machine learning: parallel and distributed approaches , 2011, KDD '11 Tutorials.

[37]  VARUN CHANDOLA,et al.  Anomaly detection: A survey , 2009, CSUR.

[38]  Xu Qian,et al.  Dimension Reduction in Intrusion Detection Using Manifold Learning , 2009, 2009 International Conference on Computational Intelligence and Security.

[39]  Xu Qian,et al.  Intrusion Detection Using Isomap and Support Vector Machine , 2009, 2009 International Conference on Artificial Intelligence and Computational Intelligence.

[40]  Christian Callegari,et al.  A Novel PCA-Based Network Anomaly Detection , 2011, 2011 IEEE International Conference on Communications (ICC).

[41]  Ronald R. Coifman,et al.  Graph Laplacian Tomography From Unknown Random Projections , 2008, IEEE Transactions on Image Processing.

[42]  Wolfgang Banzhaf,et al.  The use of computational intelligence in intrusion detection systems: A review , 2010, Appl. Soft Comput..

[43]  Ronald R. Coifman,et al.  Hierarchical Clustering Via Localized Diffusion Folders , 2010, AAAI Fall Symposium: Manifold Learning and Its Applications.

[44]  Karen A. Scarfone,et al.  Guide to Intrusion Detection and Prevention Systems (IDPS) , 2007 .

[45]  Ann B. Lee,et al.  Diffusion maps and coarse-graining: a unified framework for dimensionality reduction, graph partitioning, and data set parameterization , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[46]  Jung-Min Park,et al.  An overview of anomaly detection techniques: Existing solutions and latest technological trends , 2007, Comput. Networks.

[47]  Tom Fawcett,et al.  An introduction to ROC analysis , 2006, Pattern Recognit. Lett..

[48]  Krysia Broda,et al.  Symbolic knowledge extraction from trained neural networks: A sound approach , 2001, Artif. Intell..

[49]  Andrew H. Sung,et al.  Intrusion detection using an ensemble of intelligent paradigms , 2005, J. Netw. Comput. Appl..

[50]  Lu Li,et al.  The Application of Genetic Algorithm to Intrusion Detection in MP2P Network , 2012, ICSI.

[51]  Artur S. d'Avila Garcez,et al.  SOAR — Sparse Oracle-based Adaptive Rule extraction: Knowledge extraction from large-scale datasets to detect credit card fraud , 2010, The 2010 International Joint Conference on Neural Networks (IJCNN).

[52]  Michel Verleysen,et al.  Nonlinear Dimensionality Reduction , 2021, Computer Vision.

[53]  B. Nadler,et al.  Diffusion maps, spectral clustering and reaction coordinates of dynamical systems , 2005, math/0503445.

[54]  Jude W. Shavlik,et al.  in Advances in Neural Information Processing , 1996 .

[55]  Aloysius K. Mok,et al.  Advanced Allergy Attacks: Does a Corpus Really Help? , 2007, RAID.

[56]  Tuomo Sipola,et al.  Dimensionality Reduction Framework for Detecting Anomalies from Network Logs , 2012 .

[57]  Anil K. Jain,et al.  Algorithms for Clustering Data , 1988 .

[58]  Shawn Ostermann,et al.  Detecting Anomalous Network Traffic with Self-organizing Maps , 2003, RAID.

[59]  M Damashek,et al.  Gauging Similarity with n-Grams: Language-Independent Categorization of Text , 1995, Science.

[60]  Alok Aggarwal,et al.  Composing Signatures for Misuse Intrusion Detection System Using Genetic Algorithm in an Offline Environment , 2012, ACITY.

[61]  Bart Baesens,et al.  Rule Extraction from Support Vector Machines: An Overview of Issues and Application in Credit Scoring , 2008, Rule Extraction from Support Vector Machines.

[62]  Bart Baesens,et al.  Using Rule Extraction to Improve the Comprehensibility of Predictive Models , 2006 .

[63]  Gil David,et al.  Hierarchical data organization , clustering and denoising via localized diffusion folders , 2011 .

[64]  Andrew P. Bradley,et al.  Rule extraction from support vector machines: A review , 2010, Neurocomputing.