Analysis of network traffic features for anomaly detection

Anomaly detection in communication networks provides the basis for the uncovering of novel attacks, misconfigurations and network failures. Resource constraints for data storage, transmission and processing make it beneficial to restrict input data to features that are (a) highly relevant for the detection task and (b) easily derivable from network observations without expensive operations. Removing strong correlated, redundant and irrelevant features also improves the detection quality for many algorithms that are based on learning techniques. In this paper we address the feature selection problem for network traffic based anomaly detection. We propose a multi-stage feature selection method using filters and stepwise regression wrappers. Our analysis is based on 41 widely-adopted traffic features that are presented in several commonly used traffic data sets. With our combined feature selection method we could reduce the original feature vectors from 41 to only 16 features. We tested our results with five fundamentally different classifiers, observing no significant reduction of the detection performance. In order to quantify the practical benefits of our results, we analyzed the costs for generating individual features from standard IP Flow Information Export records, available at many routers. We show that we can eliminate 13 very costly features and thus reducing the computational effort for on-line feature generation from live traffic observations at network nodes.

[1]  Choo-Yee Ting,et al.  A Feature Selection Approach for Network Intrusion Detection , 2009, 2009 International Conference on Information Management and Engineering.

[2]  Bianca Zadrozny,et al.  Obtaining calibrated probability estimates from decision trees and naive Bayesian classifiers , 2001, ICML.

[3]  David J. Hand,et al.  Measuring classifier performance: a coherent alternative to the area under the ROC curve , 2009, Machine Learning.

[4]  Hans-Peter Kriegel,et al.  A survey on unsupervised outlier detection in high‐dimensional numerical data , 2012, Stat. Anal. Data Min..

[5]  David D. Lewis,et al.  Naive (Bayes) at Forty: The Independence Assumption in Information Retrieval , 1998, ECML.

[6]  R. Samworth.  Optimal weighted nearest neighbour classifiers , 2011, 1101.5783.

[7]  Pat Langley,et al.  Selection of Relevant Features and Examples in Machine Learning , 1997, Artif. Intell..

[8]  Jugal K. Kalita,et al.  Network Anomaly Detection: Methods, Systems and Tools , 2014, IEEE Communications Surveys & Tutorials.

[9]  Ajith Abraham,et al.  Feature deduction and ensemble design of intrusion detection systems , 2005, Comput. Secur..

[10]  Slobodan Petrovic,et al.  Towards a Generic Feature-Selection Measure for Intrusion Detection , 2010, 2010 20th International Conference on Pattern Recognition.

[11]  Ron Kohavi,et al.  Wrappers for Feature Subset Selection , 1997, Artif. Intell..

[12]  R. Tibshirani,et al.  Significance analysis of microarrays applied to the ionizing radiation response , 2001, Proceedings of the National Academy of Sciences of the United States of America.

[13]  Benoit Claise,et al.  Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of IP Traffic Flow Information , 2008, RFC.

[14]  J. Ross Quinlan,et al.  Induction of Decision Trees , 1986, Machine Learning.

[15]  R. Tibshirani,et al.  Least angle regression , 2004, math/0406456.

[16]  Ali A. Ghorbani,et al.  A detailed analysis of the KDD CUP 99 data set , 2009, 2009 IEEE Symposium on Computational Intelligence for Security and Defense Applications.

[17]  Adam Prügel-Bennett,et al.  Data mining approaches for network intrusion detection: from dimensionality reduction to misuse and anomaly detection , 2012 .

[18]  Bin Wen,et al.  Principal Component Analysis of Network Security Data Based on Projection Pursuit , 2012 .

[19]  Ali A. Ghorbani,et al.  IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS—PART C: APPLICATIONS AND REVIEWS 1 Toward Credible Evaluation of Anomaly-Based Intrusion-Detection Methods , 2022 .

[20]  A.H. Sung,et al.  Identifying important features for intrusion detection using support vector machines and neural networks , 2003, 2003 Symposium on Applications and the Internet, 2003. Proceedings..

[21]  N. Meinshausen,et al.  Stability selection , 2008, 0809.2932.

[22]  Martin A. Riedmiller,et al.  Advanced supervised learning in multi-layer perceptrons — From backpropagation to adaptive learning algorithms , 1994 .

[23]  Kien A. Hua,et al.  Decision tree classifier for network intrusion detection with GA-based feature selection , 2005, ACM Southeast Regional Conference.

[24]  Brian Trammell,et al.  Bidirectional Flow Export Using IP Flow Information Export (IPFIX) , 2008, RFC.

[25]  Manas Ranjan Patra,et al.  Network intrusion detection system: A machine learning approach , 2011, Intell. Decis. Technol..

[26]  Tso-Jung Yen,et al.  Discussion on "Stability Selection" by Meinshausen and Buhlmann , 2010 .

[27]  Sayan Mukherjee,et al.  Choosing Multiple Parameters for Support Vector Machines , 2002, Machine Learning.

[28]  Marina Thottan,et al.  Anomaly detection in IP networks , 2003, IEEE Trans. Signal Process..

[29]  Heba F. Eid,et al.  Hybrid Intelligent Intrusion Detection Scheme , 2011 .

[30]  Andrew H. Sung,et al.  The Feature Selection and Intrusion Detection Problems , 2004, ASIAN.

[31]  Shahrzad Zargari,et al.  Feature Selection in the Corrected KDD-dataset , 2012, 2012 Third International Conference on Emerging Intelligent Data and Web Technologies.

[32]  Azween Abdullah,et al.  Intrusions Detection based on Optimum Features Subset and Efficient Dataset Selection , 2012 .

[33]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[34]  Isabelle Guyon,et al.  An Introduction to Variable and Feature Selection , 2003, J. Mach. Learn. Res..

[35]  Thorsten Joachims,et al.  Text Categorization with Support Vector Machines: Learning with Many Relevant Features , 1998, ECML.

[36]  Chris H. Q. Ding,et al.  Minimum redundancy feature selection from microarray gene expression data , 2003, Computational Systems Bioinformatics. CSB2003. Proceedings of the 2003 IEEE Bioinformatics Conference. CSB2003.

[37]  Tom Fawcett,et al.  An introduction to ROC analysis , 2006, Pattern Recognit. Lett..

[38]  John McHugh,et al.  Testing Intrusion detection systems: a critique of the 1998 and 1999 DARPA intrusion detection system evaluations as performed by Lincoln Laboratory , 2000, TSEC.

[39]  Benoit Claise,et al.  Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of Flow Information , 2013, RFC.