From Intrusion Detection to Attacker Attribution: A Comprehensive Survey of Unsupervised Methods

Over the last five years there has been an increase in the frequency and diversity of network attacks. This holds true, as more and more organizations admit compromises on a daily basis. Many misuse and anomaly based intrusion detection systems (IDSs) that rely on either signatures, supervised or statistical methods have been proposed in the literature, but their trustworthiness is debatable. Moreover, as this paper uncovers, the current IDSs are based on obsolete attack classes that do not reflect the current attack trends. For these reasons, this paper provides a comprehensive overview of unsupervised and hybrid methods for intrusion detection, discussing their potential in the domain. We also present and highlight the importance of feature engineering techniques that have been proposed for intrusion detection. Furthermore, we discuss that current IDSs should evolve from simple detection to correlation and attribution. We descant how IDS data could be used to reconstruct and correlate attacks to identify attackers, with the use of advanced data analytics techniques. Finally, we argue how the present IDS attack classes can be extended to match the modern attacks and propose three new classes regarding the outgoing network communication.

[1]  Leonid Portnoy,et al.  Intrusion detection with unlabeled data using clustering , 2000 .

[2]  Vern Paxson,et al.  Measuring Pay-per-Install: The Commoditization of Malware Distribution , 2011, USENIX Security Symposium.

[3]  Zahir Tari,et al.  Toward an efficient and scalable feature selection approach for internet traffic classification , 2013, Comput. Networks.

[4]  Wang Qian,et al.  A graph-based clustering algorithm for anomaly intrusion detection , 2012, 2012 7th International Conference on Computer Science & Education (ICCSE).

[5]  Ulf Lindqvist,et al.  Modeling multistep cyber attacks for scenario recognition , 2003, Proceedings DARPA Information Survivability Conference and Exposition.

[6]  Ying Chen,et al.  Hybrid Intrusion Detection with Weighted Signature Generation over Anomalous Internet Episodes , 2007, IEEE Transactions on Dependable and Secure Computing.

[7]  Marc Dacier,et al.  Towards a taxonomy of intrusion-detection systems , 1999, Comput. Networks.

[8]  Arshad Iqbal,et al.  Network Traffic Analysis and Intrusion Detection Using Packet Sniffer , 2010, 2010 Second International Conference on Communication Software and Networks.

[9]  Ing-Ray Chen,et al.  A survey of intrusion detection in wireless network applications , 2014, Comput. Commun..

[10]  Zahir Tari,et al.  An optimal and stable feature selection approach for traffic classification based on multi-criterion fusion , 2014, Future Gener. Comput. Syst..

[11]  David M. Nicol,et al.  The Koobface botnet and the rise of social malware , 2010, 2010 5th International Conference on Malicious and Unwanted Software.

[12]  Yuan-Cheng Lai,et al.  Evasion Techniques: Sneaking through Your Intrusion Detection/Prevention Systems , 2012, IEEE Communications Surveys & Tutorials.

[13]  Ali A. Ghorbani,et al.  A detailed analysis of the KDD CUP 99 data set , 2009, 2009 IEEE Symposium on Computational Intelligence for Security and Defense Applications.

[14]  Ravi Sankar,et al.  A Survey of Intrusion Detection Systems in Wireless Sensor Networks , 2014, IEEE Communications Surveys & Tutorials.

[15]  Elsayed A. Sallam,et al.  A hybrid network intrusion detection framework based on random forests and weighted k-means , 2013 .

[16]  EMMANOUIL VASILOMANOLAKIS,et al.  Taxonomy and Survey of Collaborative Intrusion Detection , 2015, ACM Comput. Surv..

[17]  Sushil Jajodia,et al.  Correlating intrusion events and building attack scenarios through attack graph distances , 2004, 20th Annual Computer Security Applications Conference.

[18]  Gisung Kim,et al.  A novel hybrid intrusion detection method integrating anomaly detection with misuse detection , 2014, Expert Syst. Appl..

[19]  A Min Tjoa,et al.  First International Conference on Availability, Reliability and Security (ARES´06) , 2006 .

[20]  Vijay Varadharajan,et al.  Intrusion detection techniques in cloud environment: A survey , 2017, J. Netw. Comput. Appl..

[21]  S. E. Smaha.  Haystack: an intrusion detection system , 1988, [Proceedings 1988] Fourth Aerospace Computer Security Applications.

[22]  F. Dressler,et al.  Survey of Event Correlation Techniques for Attack Detection in Early Warning Systems , 2008 .

[23]  Jugal K. Kalita,et al.  A multi-step outlier-based anomaly detection approach to network-wide traffic , 2016, Inf. Sci..

[24]  Kimmo Hätönen,et al.  A computer host-based user anomaly detection system using the self-organizing map , 2000, Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium.

[25]  M.I. Heywood,et al.  Host-based intrusion detection using self-organizing maps , 2002, Proceedings of the 2002 International Joint Conference on Neural Networks. IJCNN'02 (Cat. No.02CH37290).

[26]  Jaideep Srivastava,et al.  Intrusion Detection: A Survey , 2005 .

[27]  Yu-Lin He,et al.  Fuzziness based semi-supervised learning approach for intrusion detection system , 2017, Inf. Sci..

[28]  Salvatore J. Stolfo,et al.  Data Mining Approaches for Intrusion Detection , 1998, USENIX Security Symposium.

[29]  Erhan Guven,et al.  A Survey of Data Mining and Machine Learning Methods for Cyber Security Intrusion Detection , 2016, IEEE Communications Surveys & Tutorials.

[30]  K. Raghuveer,et al.  Intrusion detection technique by using k-means, fuzzy neural network and SVM classifiers , 2013, 2013 International Conference on Computer Communication and Informatics.

[31]  Siyang Zhang,et al.  A novel hybrid KPCA and SVM with GA model for intrusion detection , 2014, Appl. Soft Comput..

[32]  Tom Fawcett,et al.  An introduction to ROC analysis , 2006, Pattern Recognit. Lett..

[33]  Antonio Martínez-Álvarez,et al.  Feature selection by multi-objective optimisation: Application to network anomaly detection by hierarchical self-organising maps , 2014, Knowl. Based Syst..

[34]  Zhen Liu,et al.  A class-oriented feature selection approach for multi-class imbalanced network traffic datasets based on local and global metrics fusion , 2015, Neurocomputing.

[35]  Ferat Sahin,et al.  A survey on feature selection methods , 2014, Comput. Electr. Eng..

[36]  Zhong Ming,et al.  An improved NSGA-III algorithm for feature selection used in intrusion detection , 2017, Knowl. Based Syst..

[37]  David M. W. Powers,et al.  Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation , 2011, ArXiv.

[38]  Jugal K. Kalita,et al.  An effective unsupervised network anomaly detection method , 2012, ICACCI '12.

[39]  Philippe Owezarski,et al.  Knowledge-independent traffic monitoring: Unsupervised detection of network attacks , 2012, IEEE Network.

[40]  Raj Acharya,et al.  An immune inspired unsupervised intrusion detection system for detection of novel attacks , 2016, 2016 IEEE Conference on Intelligence and Security Informatics (ISI).

[41]  Paul England,et al.  The Darknet and the Future of Content Distribution , 2003 .

[42]  István Szabó,et al.  On the Validation of Traffic Classification Algorithms , 2008, PAM.

[43]  Jugal K. Kalita,et al.  NADO: network anomaly detection using outlier approach , 2011, ICCCS '11.

[44]  Gabriel Maciá-Fernández,et al.  Anomaly-based network intrusion detection: Techniques, systems and challenges , 2009, Comput. Secur..

[45]  Anja Feldmann,et al.  Operational experiences with high-volume network intrusion detection , 2004, CCS '04.

[46]  Timo D. Hämäläinen,et al.  Artificial Immune System Based Intrusion Detection: Innate Immunity using an Unsupervised Learning Approach , 2014 .

[47]  Nasser Yazdani,et al.  Mutual information-based feature selection for intrusion detection systems , 2011, J. Netw. Comput. Appl..

[48]  Kalyanmoy Deb,et al.  A fast and elitist multiobjective genetic algorithm: NSGA-II , 2002, IEEE Trans. Evol. Comput..

[49]  Ali A. Ghorbani,et al.  Toward developing a systematic approach to generate benchmark datasets for intrusion detection , 2012, Comput. Secur..

[50]  Lilian Mitrou,et al.  Smartphone sensor data as digital evidence , 2013, Comput. Secur..

[51]  Dimitris Gritzalis,et al.  The Big Four - What We Did Wrong in Advanced Persistent Threat Detection? , 2013, 2013 International Conference on Availability, Reliability and Security.

[52]  Richard J. Enbody,et al.  Cybercrime: Dissecting the State of Underground Enterprise , 2013, IEEE Internet Computing.

[53]  Hervé Debar,et al.  Correlation of Intrusion Symptoms: An Application of Chronicles , 2003, RAID.

[54]  Gang Lu,et al.  Feature selection for optimizing traffic classification , 2012, Comput. Commun..

[55]  Jugal K. Kalita,et al.  MLH-IDS: A Multi-Level Hybrid Intrusion Detection Method , 2014, Comput. J..

[56]  G. Cybenko,et al.  Temporal and spatial distributed event correlation for network security , 2004, Proceedings of the 2004 American Control Conference.

[57]  Jiankun Hu,et al.  A Semantic Approach to Host-Based Intrusion Detection Systems Using Contiguousand Discontiguous System Call Patterns , 2014, IEEE Transactions on Computers.

[58]  Mohammad Zulkernine,et al.  A hybrid network intrusion detection technique using random forests , 2006, First International Conference on Availability, Reliability and Security (ARES'06).

[59]  Zhu Han,et al.  Applications of Economic and Pricing Models for Wireless Network Security: A Survey , 2017, IEEE Communications Surveys & Tutorials.

[60]  Hiroki Takakura,et al.  Toward a more practical unsupervised anomaly detection system , 2013, Inf. Sci..

[61]  Dongjoon Kong,et al.  A differentiated one-class classification method with applications to intrusion detection , 2012, Expert Syst. Appl..

[62]  Thorsten Holz,et al.  As the net churns: Fast-flux botnet observations , 2008, 2008 3rd International Conference on Malicious and Unwanted Software (MALWARE).

[63]  Christopher Krügel,et al.  Decentralized Event Correlation for Intrusion Detection , 2001, ICISC.

[64]  Rui Xu,et al.  Survey of clustering algorithms , 2005, IEEE Transactions on Neural Networks.

[65]  Ibrahim Aljarah,et al.  MapReduce intrusion detection system based on a particle swarm optimization clustering algorithm , 2013, 2013 IEEE Congress on Evolutionary Computation.

[66]  Walid Saad,et al.  Machine Learning for Wireless Networks with Artificial Intelligence: A Tutorial on Neural Networks , 2017, ArXiv.

[67]  Hwee Pink Tan,et al.  Machine Learning in Wireless Sensor Networks: Algorithms, Strategies, and Applications , 2014, IEEE Communications Surveys & Tutorials.

[68]  Payam Vahdani Amoli,et al.  Unsupervised network intrusion detection systems for zero-day fast-spreading network attacks and botnets , 2015 .

[69]  S. Sastry,et al.  SCADA-specific Intrusion Detection / Prevention Systems : A Survey and Taxonomy , 2010 .

[70]  Eric Michael Hutchins,et al.  Intelligence-Driven Computer Network Defense Informed by Analysis of Adversary Campaigns and Intrusion Kill Chains , 2010 .

[71]  VARUN CHANDOLA,et al.  Anomaly detection: A survey , 2009, CSUR.

[72]  Clayton R. Pereira,et al.  A nature-inspired approach to speed up optimum-path forest clustering and its application to intrusion detection in computer networks , 2015, Inf. Sci..

[73]  Emin Anarim,et al.  An intelligent intrusion detection system (IDS) for anomaly and misuse detection in computer networks , 2005, Expert Syst. Appl..

[74]  Yuxin Ding,et al.  Host-based intrusion detection using dynamic and static behavioral models , 2003, Pattern Recognit..

[75]  Ali A. Ghorbani,et al.  Detecting P2P botnets through network behavior analysis and machine learning , 2011, 2011 Ninth Annual International Conference on Privacy, Security and Trust.

[76]  Anna Sperotto,et al.  Flow-based intrusion detection , 2011, 12th IFIP/IEEE International Symposium on Integrated Network Management (IM 2011) and Workshops.

[77]  Xinghuo Yu,et al.  A simple and efficient hidden Markov model scheme for host-based anomaly intrusion detection , 2009, IEEE Network.

[78]  Tareek M. Pattewar,et al.  Neural network based intrusion detection using Bayesian with PCA and KPCA feature extraction , 2015, 2015 IEEE International Conference on Computer Graphics, Vision and Information Security (CGVIS).

[79]  Vern Paxson,et al.  Bro: a system for detecting network intruders in real-time , 1998, Comput. Networks.

[80]  Hari Om,et al.  A hybrid system for reducing the false alarm rate of anomaly intrusion detection system , 2012, 2012 1st International Conference on Recent Advances in Information Technology (RAIT).

[81]  Christin Schäfer,et al.  Learning Intrusion Detection: Supervised or Unsupervised? , 2005, ICIAP.

[82]  Andrew W. Moore,et al.  Architecture of a network monitor , 2003 .

[83]  Jugal K. Kalita,et al.  Network Anomaly Detection: Methods, Systems and Tools , 2014, IEEE Communications Surveys & Tutorials.

[84]  Neelam Dwivedi,et al.  Event Correlation for Intrusion Detection Systems , 2015, 2015 IEEE International Conference on Computational Intelligence & Communication Technology.

[85]  Philippe Owezarski,et al.  Unsupervised Network Intrusion Detection Systems: Detecting the Unknown without Knowledge , 2012, Comput. Commun..

[86]  Carlos García Garino,et al.  Automatic network intrusion detection: Current techniques and open issues , 2012, Comput. Electr. Eng..

[87]  Mansour Sheikhan,et al.  Modification of supervised OPF-based intrusion detection systems using unsupervised learning and social network concept , 2017, Pattern Recognit..

[88]  Andrew W. Moore,et al.  Discriminators for use in flow-based classification , 2013 .

[89]  Akira Kato,et al.  Traffic Data Repository at the WIDE Project , 2000, USENIX Annual Technical Conference, FREENIX Track.

[90]  Sushil Jajodia,et al.  ADAM: Detecting Intrusions by Data Mining , 2001 .

[91]  Chih-Fong Tsai,et al.  CANN: An intrusion detection system based on combining cluster centers and nearest neighbors , 2015, Knowl. Based Syst..

[92]  Yang Li,et al.  Building lightweight intrusion detection system using wrapper-based feature selection mechanisms , 2009, Comput. Secur..

[93]  Alfonso Valdes,et al.  Next-generation Intrusion Detection Expert System (NIDES)A Summary , 1997 .

[94]  Walid Saad,et al.  Device Fingerprinting in Wireless Networks: Challenges and Opportunities , 2015, IEEE Communications Surveys & Tutorials.

[95]  Giovanni Vigna,et al.  Intrusion detection: a brief history and overview , 2002 .

[96]  William H. Sanders,et al.  Intrusion detection in enterprise systems by combining and clustering diverse monitor data , 2016, HotSoS.

[97]  Volker Roth,et al.  Feature Selection in Clustering Problems , 2003, NIPS.

[98]  Chun-Hung Richard Lin,et al.  Intrusion detection system: A comprehensive review , 2013, J. Netw. Comput. Appl..

[99]  Richard Frank,et al.  Identifying digital threats in a hacker web forum , 2015, 2015 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM).