Network Environment Design for Autonomous Cyberdefense

Reinforcement learning (RL) has been demonstrated suitable to develop agents that play complex games with human-level performance. However, it is not understood how to effectively use RL to perform cybersecurity tasks. To develop such understanding, it is necessary to develop RL agents using simulation and emulation systems allowing researchers to model a broad class of realistic threats and network conditions. Demonstrating that a specific RL algorithm can be effective for defending a network under certain conditions may not necessarily give insight about the performance of the algorithm when the threats, network conditions, and security goals change. This paper introduces a novel approach for network environment design and a software framework to address the fundamental problem that network defense cannot be defined as a single game with a simple set of fixed rules. We show how our approach is necessary to facilitate the development of RL network defenders that are robust against attacks aimed at the agent’s learning. Our framework enables the development and simulation of adversaries with sophisticated behavior that includes poisoning and evasion attacks on RL network defenders.

[1]  Vikash K. Mansinghka,et al.  Gen: a general-purpose probabilistic programming system with programmable inference , 2019, PLDI.

[2]  Shimon Whiteson,et al.  The StarCraft Multi-Agent Challenge , 2019, AAMAS.

[3]  Yang Song,et al.  An Overview of Microsoft Academic Service (MAS) and Applications , 2015, WWW.

[4]  Dawn Xiaodong Song,et al.  Delving into adversarial attacks on deep policies , 2017, ICLR.

[5]  Jonathon Schwartz,et al.  CybORG: An Autonomous Cyber Operations Research Gym , 2020, ArXiv.

[6]  Ivan Beschastnikh,et al.  Iroko: A Framework to Prototype Reinforcement Learning for Data Center Traffic Control , 2018, ArXiv.

[7]  Marc G. Bellemare,et al.  The Arcade Learning Environment: An Evaluation Platform for General Agents , 2012, J. Artif. Intell. Res..

[8]  Michael I. Jordan,et al.  Ray: A Distributed Framework for Emerging AI Applications , 2017, OSDI.

[9]  Sergey Levine,et al.  Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design , 2020, NeurIPS.

[10]  Kian Hsiang Low,et al.  Inverse Reinforcement Learning with Locally Consistent Reward Functions , 2015, NIPS.

[11]  Ion Stoica,et al.  Ray RLLib: A Composable and Scalable Reinforcement Learning Library , 2017, NIPS 2017.

[12]  Feng Liu,et al.  AuTO: scaling deep reinforcement learning for datacenter-scale automatic traffic optimization , 2018, SIGCOMM.

[13]  Christopher Leckie,et al.  Reinforcement Learning for Autonomous Defence in Software-Defined Networking , 2018, GameSec.

[14]  Katja Hofmann,et al.  The Malmo Platform for Artificial Intelligence Experimentation , 2016, IJCAI.

[15]  Bob Lantz,et al.  A Mininet-based Virtual Testbed for Distributed SDN Development , 2015, Comput. Commun. Rev..

[16]  Christopher Leckie,et al.  Adversarial Reinforcement Learning under Partial Observability in Software-Defined Networking , 2019, ArXiv.

[17]  Vijay Janapa Reddi,et al.  Deep Reinforcement Learning for Cyber Security , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[18]  Wojciech Zaremba,et al.  OpenAI Gym , 2016, ArXiv.

[19]  Alex Graves,et al.  Playing Atari with Deep Reinforcement Learning , 2013, ArXiv.

[20]  Holger Karl,et al.  MeDICINE: Rapid prototyping of production-ready network services in multi-PoP environments , 2016, 2016 IEEE Conference on Network Function Virtualization and Software Defined Networks (NFV-SDN).

[21]  Marcin Andrychowicz,et al.  Solving Rubik's Cube with a Robot Hand , 2019, ArXiv.

[22]  Tom Schaul,et al.  StarCraft II: A New Challenge for Reinforcement Learning , 2017, ArXiv.

[23]  Sergey Levine,et al.  Adversarial Policies: Attacking Deep Reinforcement Learning , 2019, ICLR.

[24]  Arslan Munir,et al.  The Faults in Our Pi Stars: Security Issues and Open Challenges in Deep Reinforcement Learning , 2018, ArXiv.

[25]  Yuxi Li,et al.  Deep Reinforcement Learning , 2018, Reinforcement Learning for Cyber-Physical Systems.