Investigating Deep Learning for Collective Anomaly Detection - An Experimental Study

This paper explores the effectiveness of deep learning and other supervised learning algorithms for collective anomaly detection. Almost all the approaches so far proposed for DoS (Denial of Service) attack detection with the aid of collective anomaly detection are unsupervised in nature. Due to this reason, often those approaches show high false alarm rates. To reduce the high false alarm rate, we have done some experiments to investigate the suitability of deep learning for this field. Interestingly, the obtained experimental results on UNSW-NB15 and KDD Cup 1999 datasets show that the deep learning implemented using H2O achieves approximately 97% recall for collective anomaly detection. Hence, deep learning outperforms a wide range of unsupervised techniques for collective anomaly detection. This is the first reported work that investigates collective anomaly detection problem using deep learning.

[1]  Ali A. Ghorbani,et al.  A detailed analysis of the KDD CUP 99 data set , 2009, 2009 IEEE Symposium on Computational Intelligence for Security and Defense Applications.

[2]  Mohiuddin Ahmed,et al.  Collective Anomaly Detection Techniques for Network Traffic Analysis , 2018 .

[3]  Bernhard Schölkopf,et al.  One-Class Support Measure Machines for Group Anomaly Detection , 2013, UAI.

[4]  Dong Yu,et al.  Deep Learning: Methods and Applications , 2014, Found. Trends Signal Process..

[5]  Mohiuddin Ahmed,et al.  Network Traffic Pattern Analysis Using Improved Information Theoretic Co-clustering Based Collective Anomaly Detection , 2014, SecureComm.

[6]  Mohiuddin Ahmed,et al.  Network traffic analysis based on collective anomaly detection , 2014, 2014 9th IEEE Conference on Industrial Electronics and Applications.

[7]  Mia Hubert,et al.  Clustering in an object-oriented environment , 1997 .

[8]  Yan Liu,et al.  GLAD: group anomaly detection in social media analysis , 2014, KDD.

[9]  Hans-Peter Kriegel,et al.  LOF: identifying density-based local outliers , 2000, SIGMOD 2000.

[10]  Mohiuddin Ahmed,et al.  A survey of network anomaly detection techniques , 2016, J. Netw. Comput. Appl..

[11]  Zhilin Li,et al.  A Multiscale Approach for Spatio‐Temporal Outlier Detection , 2006, Trans. GIS.

[12]  Mohiuddin Ahmed,et al.  Anomaly Detection on Big Data in Financial Markets , 2017, 2017 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM).

[13]  Jill Slay,et al.  The evaluation of Network Anomaly Detection Systems: Statistical analysis of the UNSW-NB15 data set and the comparison with the KDD99 data set , 2016, Inf. Secur. J. A Glob. Perspect..

[14]  Rajeev Rastogi,et al.  Efficient algorithms for mining outliers from large data sets , 2000, SIGMOD 2000.

[15]  Mohiuddin Ahmed,et al.  Thwarting DoS Attacks: A Framework for Detection based on Collective Anomalies and Clustering , 2017, Computer.

[16]  Mohiuddin Ahmed,et al.  Novel Approach for Network Traffic Pattern Analysis using Clustering-based Collective Anomaly Detection , 2015, Annals of Data Science.

[17]  Christopher Leckie,et al.  Unsupervised Anomaly Detection in Network Intrusion Detection Using Clusters , 2005, ACSC.