An efficient reinforcement learning-based Botnet detection approach

Abstract The use of bot malware and botnets as a tool to facilitate other malicious cyber activities (e.g. distributed denial of service attacks, dissemination of malware and spam, and click fraud). However, detection of botnets, particularly peer-to-peer (P2P) botnets, is challenging. Hence, in this paper we propose a sophisticated traffic reduction mechanism, integrated with a reinforcement learning technique. We then evaluate the proposed approach using real-world network traffic, and achieve a detection rate of 98.3%. The approach also achieves a relatively low false positive rate (i.e. 0.012%).

[1]  Tahar Kechadi,et al.  Peer-to-Peer Botnet Investigation: A Review , 2012 .

[2]  Michael K. Reiter,et al.  Traffic Aggregation for Malware Detection , 2008, DIMVA.

[3]  B. B. Gupta,et al.  A Survey of Phishing Email Filtering Techniques , 2013, IEEE Communications Surveys & Tutorials.

[4]  Paul Rad,et al.  Implementation of deep packet inspection in smart grids and industrial Internet of Things: Challenges and opportunities , 2019, J. Netw. Comput. Appl..

[5]  Ali A. Ghorbani,et al.  Group Behavior Metrics for P2P Botnet Detection , 2012, ICICS.

[6]  Song Guo,et al.  Can We Beat DDoS Attacks in Clouds? , 2014, IEEE Transactions on Parallel and Distributed Systems.

[7]  Xiaoju Dong,et al.  Research on Visualization Systems for DDoS Attack Detection , 2018, 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC).

[8]  Matthew Roughan,et al.  Class-of-service mapping for QoS: a statistical signature-based approach to IP traffic classification , 2004, IMC '04.

[9]  Brent Byunghoon Kang,et al.  Peer-to-Peer Botnets: Overview and Case Study , 2007, HotBots.

[10]  Rhiannon Weaver,et al.  A Probabilistic Population Study of the Conficker-C Botnet , 2010, PAM.

[11]  Andreas Hotho,et al.  A Survey of Network-based Intrusion Detection Data Sets , 2019, Comput. Secur..

[12]  José M. Fernandez,et al.  Optimising sybil attacks against P2P-based botnets , 2009, 2009 4th International Conference on Malicious and Unwanted Software (MALWARE).

[13]  W. Timothy Strayer,et al.  Botnet Detection Based on Network Behavior , 2008, Botnet Detection.

[14]  Sven Dietrich,et al.  P2P as botnet command and control: A deeper insight , 2008, 2008 3rd International Conference on Malicious and Unwanted Software (MALWARE).

[15]  Ting-Fang Yen,et al.  Detecting Stealthy Malware Using Behavioral Features in Network Traffic , 2011 .

[16]  Jiankun Hu,et al.  A holistic review of Network Anomaly Detection Systems: A comprehensive survey , 2019, J. Netw. Comput. Appl..

[17]  Mouhammd Alkasassbeh,et al.  An empirical evaluation for the intrusion detection features based on machine learning and feature selection methods , 2017, ArXiv.

[18]  Nina Taft,et al.  Passive and Active Measurement , 2012, Lecture Notes in Computer Science.

[19]  Wei Jiang,et al.  Botnet: Survey and Case Study , 2009, 2009 Fourth International Conference on Innovative Computing, Information and Control (ICICIC).

[20]  Song Guo,et al.  Discriminating DDoS Attacks from Flash Crowds Using Flow Correlation Coefficient , 2012, IEEE Transactions on Parallel and Distributed Systems.

[21]  Sanmeet Kaur,et al.  Issues and challenges in DNS based botnet detection: A survey , 2019, Comput. Secur..

[22]  Wen-Hwa Liao,et al.  Peer to Peer Botnet Detection Using Data Mining Scheme , 2010, 2010 International Conference on Internet Technology and Applications.

[23]  Thorsten Holz,et al.  Rishi: Identify Bot Contaminated Hosts by IRC Nickname Evaluation , 2007, HotBots.

[24]  Kevin W. Hamlen,et al.  Flow-based identification of botnet traffic by mining multiple log files , 2008, 2008 First International Conference on Distributed Framework and Applications.

[25]  Albert Nigrin,et al.  Neural networks for pattern recognition , 1993 .

[26]  Zyad Shaaban,et al.  Normalization as a Preprocessing Engine for Data Mining and the Approach of Preference Matrix , 2006, 2006 International Conference on Dependability of Computer Systems.

[27]  Chun-Ying Huang,et al.  A fuzzy pattern-based filtering algorithm for botnet detection , 2011, Comput. Networks.

[28]  Xiuli Shao,et al.  Detecting P2P botnets by discovering flow dependency in C&C traffic , 2014, Peer-to-Peer Netw. Appl..

[29]  W. Timothy Strayer,et al.  Using Machine Learning Techniques to Identify Botnet Traffic , 2006 .

[30]  Ian H. Witten,et al.  Data mining: practical machine learning tools and techniques, 3rd Edition , 1999 .

[31]  Wei-Yin Loh,et al.  Classification and regression trees , 2011, WIREs Data Mining Knowl. Discov..

[32]  Ronaldo M. Salles,et al.  Botnets: A survey , 2013, Comput. Networks.

[33]  Slobodan Petrovic,et al.  A Comparison of Feature-Selection Methods for Intrusion Detection , 2010, MMM-ACNS.

[34]  Yun Yang,et al.  Research on P2P Botnet Network Behaviors and Modeling , 2012, ICICA.

[35]  Falko Dressler,et al.  Flow-based TCP connection analysis , 2009, 2009 IEEE 28th International Performance Computing and Communications Conference.

[36]  Amr M. Youssef,et al.  On the analysis of the Zeus botnet crimeware toolkit , 2010, 2010 Eighth International Conference on Privacy, Security and Trust.

[37]  Kuriakose Athappilly,et al.  A comparative predictive analysis of neural networks (NNs), nonlinear regression and classification and regression tree (CART) models , 2005, Expert Syst. Appl..

[38]  Ting Yu,et al.  A Survey on Malicious Domains Detection through DNS Data Analysis , 2018, ACM Comput. Surv..

[39]  Maarten van Someren,et al.  A Bias-Variance Analysis of a Real World Learning Problem: The CoIL Challenge 2000 , 2004, Machine Learning.

[40]  Hieu Nguyen,et al.  Multi-Confirmations and DNS Graph Mining for Malicious Domain Detection , 2019 .

[41]  David M. J. Tax,et al.  One-class classification , 2001 .

[42]  Riccardo Gusella,et al.  Characterizing the Variability of Arrival Processes with Indexes of Dispersion , 1991, IEEE J. Sel. Areas Commun..

[43]  Vinod Yegneswaran,et al.  BotHunter: Detecting Malware Infection Through IDS-Driven Dialog Correlation , 2007, USENIX Security Symposium.

[44]  Sateesh K. Peddoju,et al.  Behaviour analysis of machine learning algorithms for detecting P2P botnets , 2013, 2013 15th International Conference on Advanced Computing Technologies (ICACT).

[45]  Ali A. Ghorbani,et al.  Clustering botnet communication traffic based on n-gram feature selection , 2011, Comput. Commun..

[46]  Sureswaran Ramadass,et al.  A Survey of Botnet and Botnet Detection , 2009, 2009 Third International Conference on Emerging Security Information, Systems and Technologies.

[47]  Felix C. Freiling,et al.  Measurements and Mitigation of Peer-to-Peer-based Botnets: A Case Study on Storm Worm , 2008, LEET.

[48]  Meikang Qiu,et al.  Reinforcement Learning-based Content-Centric Services in Mobile Sensing , 2018, IEEE Network.

[49]  Joos Vandewalle,et al.  Constructing fuzzy models with linguistic integrity from numerical data-AFRELI algorithm , 2000, IEEE Trans. Fuzzy Syst..

[50]  Anton Dries,et al.  Adaptive concept drift detection , 2009, SDM.

[51]  Christopher Krügel,et al.  On the Effectiveness of Techniques to Detect Phishing Sites , 2007, DIMVA.

[52]  Anil K. Jain,et al.  Data clustering: a review , 1999, CSUR.

[53]  Guofei Gu,et al.  BotSniffer: Detecting Botnet Command and Control Channels in Network Traffic , 2008, NDSS.

[54]  Dario Rossi,et al.  Black-box analysis of Internet P2P applications , 2011, Peer-to-Peer Netw. Appl..

[55]  Dae-il Jang,et al.  Evasion technique and detection of malicious botnet , 2010, 2010 International Conference for Internet Technology and Secured Transactions.

[56]  Chadi Barakat,et al.  Can We Trust the Inter-Packet Time for Traffic Classification? , 2011, 2011 IEEE International Conference on Communications (ICC).

[57]  Li Zhang,et al.  A P2P Botnet detection scheme based on decision tree and adaptive multilayer neural networks , 2016, Neural Computing and Applications.

[58]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[59]  Hung-Min Sun,et al.  Bot detection using unsupervised machine learning , 2018 .

[60]  Eul Gyu Im,et al.  A Survey on P2P Botnet Detection , 2011, ICITCS.

[61]  Ioannis Agrafiotis,et al.  The challenge of detecting sophisticated attacks: Insights from SOC Analysts , 2018, ARES.

[62]  Ali A. Ghorbani,et al.  Botnet detection based on traffic behavior analysis and flow intervals , 2013, Comput. Secur..

[63]  Ali A. Ghorbani,et al.  Detecting P2P botnets through network behavior analysis and machine learning , 2011, 2011 Ninth Annual International Conference on Privacy, Security and Trust.

[64]  Ali A. Ghorbani,et al.  Toward developing a systematic approach to generate benchmark datasets for intrusion detection , 2012, Comput. Secur..

[65]  Hossein Rouhani Zeidanloo,et al.  A taxonomy of Botnet detection techniques , 2010, 2010 3rd International Conference on Computer Science and Information Technology.

[66]  Ali Dehghantanha,et al.  Intelligent OS X malware threat detection with code inspection , 2018, Journal of Computer Virology and Hacking Techniques.

[67]  Tian Zhang,et al.  BIRCH: A New Data Clustering Algorithm and Its Applications , 1997, Data Mining and Knowledge Discovery.

[68]  Kyoung Soo Han,et al.  The Traffic Analysis of P2P-based Storm Botnet using Honeynet , 2009, INSCRYPT 2009.

[69]  Wang Hui,et al.  Network Data Packet Capture and Protocol Analysis on Jpcap-Based , 2009, 2009 International Conference on Information Management, Innovation Management and Industrial Engineering.

[70]  Maen Alzubi,et al.  Evaluation of machine learning algorithms for intrusion detection system , 2017, 2017 IEEE 15th International Symposium on Intelligent Systems and Informatics (SISY).

[71]  Martin A. Riedmiller,et al.  A direct adaptive method for faster backpropagation learning: the RPROP algorithm , 1993, IEEE International Conference on Neural Networks.

[72]  Tom Fawcett,et al.  An introduction to ROC analysis , 2006, Pattern Recognit. Lett..

[73]  Kang Li,et al.  PeerRush: Mining for unwanted P2P traffic , 2013, J. Inf. Secur. Appl..

[74]  Anton Dries,et al.  Adaptive concept drift detection , 2009 .

[75]  John A. Swets,et al.  Signal Detection Theory and ROC Analysis in Psychology and Diagnostics: Collected Papers , 1996 .

[76]  H. Guterman,et al.  Knowledge extraction from artificial neural network models , 1997, 1997 IEEE International Conference on Systems, Man, and Cybernetics. Computational Cybernetics and Simulation.

[77]  Guofei Gu,et al.  BotMiner: Clustering Analysis of Network Traffic for Protocol- and Structure-Independent Botnet Detection , 2008, USENIX Security Symposium.

[78]  Xiapu Luo,et al.  Detecting stealthy P2P botnets using statistical traffic fingerprints , 2011, 2011 IEEE/IFIP 41st International Conference on Dependable Systems & Networks (DSN).

[79]  Francesco Palmieri,et al.  A botnet-based command and control approach relying on swarm intelligence , 2014, J. Netw. Comput. Appl..

[80]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[81]  Jae-Seo Lee,et al.  Detecting P2P Botnets Using a Multi-phased Flow Model , 2009, 2009 Third International Conference on Digital Society.

[82]  Wei-Yang Lin,et al.  Intrusion detection by machine learning: A review , 2009, Expert Syst. Appl..

[83]  B. Matthews Comparison of the predicted and observed secondary structure of T4 phage lysozyme. , 1975, Biochimica et biophysica acta.