Detecting Stealthy Botnets in a Resource-Constrained Environment using Reinforcement Learning

Modern botnets can persist in networked systems for extended periods of time by operating in a stealthy manner. Despite the progress made in the area of botnet prevention, detection, and mitigation, stealthy botnets continue to pose a significant risk to enterprises. Furthermore, existing enterprise-scale solutions require significant resources to operate effectively, thus they are not practical. In order to address this important problem in a resource-constrained environment, we propose a reinforcement learning based approach to optimally and dynamically deploy a limited number of defensive mechanisms, namely honeypots and network-based detectors, within the target network. The ultimate goal of the proposed approach is to reduce the lifetime of stealthy botnets by maximizing the number of bots identified and taken down through a sequential decision-making process. We provide a proof-of-concept of the proposed approach, and study its performance in a simulated environment. The results show that the proposed approach is promising in protecting against stealthy botnets.

[1]  Guofei Gu,et al.  BotSniffer: Detecting Botnet Command and Control Channels in Network Traffic , 2008, NDSS.

[2]  Sushil Jajodia,et al.  Dynamic Scheduling of Cybersecurity Analysts for Minimizing Risk Using Reinforcement Learning , 2016, ACM Trans. Intell. Syst. Technol..

[3]  John R. Vacca Computer and Information Security Handbook , 2009 .

[4]  Michael P. Wellman,et al.  Empirical Game-Theoretic Analysis of an Adaptive Cyber-Defense Scenario (Preliminary Report) , 2014, GameSec.

[5]  Lei Xu,et al.  Enhancing Network Security through Software Defined Networking (SDN) , 2016, 2016 25th International Conference on Computer Communication and Networks (ICCCN).

[6]  Sushil Jajodia,et al.  A Moving Target Defense Approach to Disrupting Stealthy Botnets , 2016, MTD@CCS.

[7]  Warren B. Powell,et al.  An Optimal Approximate Dynamic Programming Algorithm for the Lagged Asset Acquisition Problem , 2009, Math. Oper. Res..

[8]  Vinod Yegneswaran,et al.  BotHunter: Detecting Malware Infection Through IDS-Driven Dialog Correlation , 2007, USENIX Security Symposium.

[9]  Sean R Eddy,et al.  What is dynamic programming? , 2004, Nature Biotechnology.

[10]  Márk Jelasity,et al.  PeerSim: A scalable P2P simulator , 2009, 2009 IEEE Ninth International Conference on Peer-to-Peer Computing.

[11]  Tansu Alpcan,et al.  A Malware Detector Placement Game for Intrusion Detection , 2007, CRITIS.

[12]  Eric Chien,et al.  W32.Duqu: The Precursor to the Next Stuxnet , 2012, LEET.

[13]  Panos M. Pardalos,et al.  Approximate dynamic programming: solving the curses of dimensionality , 2009, Optim. Methods Softw..

[14]  L. Spitzner,et al.  Honeypots: Tracking Hackers , 2002 .

[15]  Ali A. Ghorbani,et al.  Toward developing a systematic approach to generate benchmark datasets for intrusion detection , 2012, Comput. Secur..

[16]  Ananthram Swami,et al.  Optimal Monitor Placement for Detection of Persistent Threats , 2016, 2016 IEEE Global Communications Conference (GLOBECOM).

[17]  Xiapu Luo,et al.  Building a Scalable System for Stealthy P2P-Botnet Detection , 2014, IEEE Transactions on Information Forensics and Security.

[18]  Guofei Gu,et al.  BotMiner: Clustering Analysis of Network Traffic for Protocol- and Structure-Independent Botnet Detection , 2008, USENIX Security Symposium.

[19]  Xiapu Luo,et al.  Detecting stealthy P2P botnets using statistical traffic fingerprints , 2011, 2011 IEEE/IFIP 41st International Conference on Dependable Systems & Networks (DSN).

[20]  T. Başar,et al.  An Intrusion Detection Game with Limited Observations , 2005 .

[21]  Ali A. Ghorbani,et al.  Towards effective feature selection in machine learning-based botnet detection approaches , 2014, 2014 IEEE Conference on Communications and Network Security.

[22]  Herbert Bos,et al.  SoK: P2PWNED - Modeling and Evaluating the Resilience of Peer-to-Peer Botnets , 2013, 2013 IEEE Symposium on Security and Privacy.

[23]  Farnam Jahanian,et al.  A Survey of Botnet Technology and Defenses , 2009, 2009 Cybersecurity Applications & Technology Conference for Homeland Security.

[24]  Abhijit Gosavi,et al.  Simulation-Based Optimization: Parametric Optimization Techniques and Reinforcement Learning , 2003 .

[25]  Diana Bohm,et al.  Computer And Information Security Handbook , 2016 .

[26]  Abhijit Gosavi,et al.  Simulation-Based Optimization: Parametric Optimization Techniques and Reinforcement Learning , 2003 .

[27]  Yang Xiang,et al.  Modeling the Propagation of Worms in Networks: A Survey , 2014, IEEE Communications Surveys & Tutorials.