A real time unsupervised NIDS for detecting unknown and encrypted network attacks in high speed network

Previously, Network Intrusion Detection Systems (NIDS) detected intrusions by comparing the behaviour of the network to the pre-defined rules or pre-observed network traffic, which was expensive in terms of both cost and time. Unsupervised machine learning techniques have overcome these issues and can detect unknown and complex attacks within normal or encrypted communication without any prior knowledge. NIDS monitors bytes, packets and network flow to detect intrusions. It is nearly impossible to monitor the payload of all packets in a high-speed network. On the other hand, the content of packets does not have sufficient information to detect a complex attack. Since the rate of attacks within encrypted communication is increasing and the content of encrypted packets is not accessible to NIDS, it has been suggested to monitor network flows. As most network intrusions spread within the network very quickly, in this paper we will propose a new real-time unsupervised NIDS for detecting new and complex attacks within normal and encrypted communications. To achieve having a real-time NIDS, the proposed model should capture live network traffic from different sensors and analyse specific metrics such as number of bytes, packets, network flows, and the time explicitly and implicitly, of packets and network flows, in the different resolutions. The NIDS will flag the time slot as an anomaly if any of those metrics passes the threshold, and it will send the time slot to the first engine. The first engine clusters different layers and dimensions of the network's behaviour and correlates the outliers to purge the intrusions from normal traffic. Detecting network attacks, which produce a huge amount of network traffic (e.g. DOS, DDOS, scanning) was the aim of proposing the first engine. Analysing statistics of network flows increases the feasibility of detecting intrusions within encrypted communications. The aim of proposing the second engine is to conduct a deeper analysis and correlate the traffic and behaviour of Bots (current attackers) during DDOS attacks to find the Bot-Master.

[1]  F. F. Etemad,et al.  Real-time Botnet command and control characterization at the host level , 2012, 6th International Symposium on Telecommunications (IST).

[2]  Hossein Rouhani Zeidanloo,et al.  Botnet detection based on traffic monitoring , 2010, 2010 International Conference on Networking and Information Technology.

[3]  Jugal K. Kalita,et al.  An effective unsupervised network anomaly detection method , 2012, ICACCI '12.

[4]  Hossein Rouhani Zeidanloo,et al.  A taxonomy of Botnet detection techniques , 2010, 2010 3rd International Conference on Computer Science and Information Technology.

[5]  Gabi Dreo Rodosek,et al.  Security System for Encrypted Environments (S2E2) , 2010, RAID.

[6]  Philippe Owezarski,et al.  Unsupervised Network Intrusion Detection Systems: Detecting the Unknown without Knowledge , 2012, Comput. Commun..

[7]  Hiroshi Tsunoda,et al.  A Multi-Stage Network Anomaly Detection Method for Improving Efficiency and Accuracy , 2012, J. Information Security.

[8]  Brian Rexroad,et al.  Wide-Scale Botnet Detection and Characterization , 2007, HotBots.

[9]  Gabriel Maciá-Fernández,et al.  Anomaly-based network intrusion detection: Techniques, systems and challenges , 2009, Comput. Secur..

[10]  R. Karimi,et al.  NEW DETECTION TECHNIQUE USING CORRELATION OF NETWORK FLOWS FOR NIDS , 2011 .

[11]  Graham Cormode,et al.  What's new: finding significant differences in network data streams , 2004, IEEE/ACM Transactions on Networking.

[12]  Mark Crovella,et al.  Characterization of network-wide anomalies in traffic flows , 2004, IMC '04.

[13]  Benoit Claise,et al.  Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of IP Traffic Flow Information , 2008, RFC.

[14]  Hans-Peter Kriegel,et al.  A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise , 1996, KDD.

[15]  Morteza Amini,et al.  RT-UNNID: A practical solution to real-time network-based intrusion detection using unsupervised neural networks , 2006, Comput. Secur..

[16]  Terrence P. Fries,et al.  A fuzzy-genetic approach to network intrusion detection , 2008, GECCO '08.

[17]  Uwe Aickelin,et al.  An Immune Inspired Network Intrusion Detection System Utilising Correlation Context , 2009, ArXiv.

[18]  Monis Akhlaq,et al.  MARS: Multi-stage Attack Recognition System , 2010, 2010 24th IEEE International Conference on Advanced Information Networking and Applications.

[19]  Baosheng Wang,et al.  Detection Network Anomalies Based on Packet and Flow Analysis , 2008, Seventh International Conference on Networking (icn 2008).

[20]  Ali A. Ghorbani,et al.  Toward developing a systematic approach to generate benchmark datasets for intrusion detection , 2012, Comput. Secur..

[21]  Hussein A. Abbass,et al.  IEEE Symposium on Computational Intelligence for Security and Defense Applications (CISDA), 2009 , 2009 .

[22]  Vegard Engen.  Machine learning for network based intrusion detection : an investigation into discrepancies in findings with the KDD cup '99 data set and multi-objective evolution of neural network classifier ensembles from imbalanced data , 2010 .

[23]  Grenville J. Armitage,et al.  A survey of techniques for internet traffic classification using machine learning , 2008, IEEE Communications Surveys & Tutorials.

[24]  Gabi Dreo Rodosek,et al.  Command Evaluation in Encrypted Remote Sessions , 2010, 2010 Fourth International Conference on Network and System Security.

[25]  M. Augustin,et al.  Intrusion detection with early recognition of encrypted application , 2011, 2011 15th IEEE International Conference on Intelligent Engineering Systems.

[26]  Kotagiri Ramamohanarao,et al.  Proactively Detecting Distributed Denial of Service Attacks Using Source IP Address Monitoring , 2004, NETWORKING.

[27]  Ali A. Ghorbani,et al.  A detailed analysis of the KDD CUP 99 data set , 2009, 2009 IEEE Symposium on Computational Intelligence for Security and Defense Applications.

[28]  Chia-Mei Chen,et al.  Flow Based Botnet Detection , 2009, 2009 Fourth International Conference on Innovative Computing, Information and Control (ICICIC).

[29]  Hossein Rouhani Zeidanloo,et al.  All About Malwares (Malicious Codes) , 2010, Security and Management.

[30]  P. Mahalanobis.  On the generalized distance in statistics , 1936 .

[31]  Aiko Pras,et al.  An Overview of IP Flow-Based Intrusion Detection , 2010, IEEE Communications Surveys & Tutorials.