Towards Scalable Verification of RL-Driven Systems

Deep neural networks (DNNs) have gained significant popularity in recent years, becoming the state of the art in a variety of domains. In particular, deep reinforcement learning (DRL) has recently been employed to train DNNs that act as control policies for various types of real-world systems. In this work, we present the whiRL 2.0 tool, which implements a new approach for verifying complex properties of interest for such DRL systems. To demonstrate the benefits of whiRL 2.0, we apply it to case studies from the communication networks domain that have recently been used to motivate formal verification of DRL systems, and which exhibit characteristics that are conducive for scalable verification. We propose techniques for performing k-induction and automated invariant inference on such systems, and use these techniques for proving safety and liveness properties of interest that were previously impossible to verify due to the scalability barriers of prior approaches. Furthermore, we show how our proposed techniques provide insights into the inner workings and the generalizability of DRL systems. whiRL 2.0 is publicly available online.

[1]  Waleed Meleis,et al.  QTCP: Adaptive Congestion Control with Reinforcement Learning , 2019, IEEE Transactions on Network Science and Engineering.

[2]  Thomas Wahl,et al.  New Passive and Active Attacks on Deep Neural Networks in Medical Applications , 2020, 2020 IEEE/ACM International Conference On Computer Aided Design (ICCAD).

[3]  Yuedong Xu,et al.  Deep Reinforcement Learning for Multi-Resource Multi-Machine Job Scheduling , 2017, ArXiv.

[4]  Hongzi Mao,et al.  Interpreting Deep Learning-Based Networking Systems , 2019, SIGCOMM.

[5]  Clark W. Barrett,et al.  Simplifying Neural Networks Using Formal Verification , 2020, NFM.

[6]  Nham Le,et al.  Verification of Recurrent Neural Networks for Cognitive Tasks via Reachability Analysis , 2020, ECAI.

[7]  Guy Katz,et al.  Verifying Recurrent Neural Networks using Invariant Inference , 2020, ATVA.

[8]  Inderjit S. Dhillon,et al.  Towards Fast Computation of Certified Robustness for ReLU Networks , 2018, ICML.

[9]  Ke Zhou,et al.  An End-to-End Automatic Cloud Database Tuning System Using Deep Reinforcement Learning , 2019, SIGMOD Conference.

[10]  Russ Tedrake,et al.  Evaluating Robustness of Neural Networks with Mixed Integer Programming , 2017, ICLR.

[11]  Marco Canini,et al.  Analyzing Learning-Based Networked Systems with Formal Verification , 2021, IEEE INFOCOM 2021 - IEEE Conference on Computer Communications.

[12]  Mykel J. Kochenderfer,et al.  Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks , 2017, CAV.

[13]  Swarat Chaudhuri,et al.  AI2: Safety and Robustness Certification of Neural Networks with Abstract Interpretation , 2018, 2018 IEEE Symposium on Security and Privacy (SP).

[14]  Guy Katz,et al.  Verifying learning-augmented systems , 2021, SIGCOMM.

[15]  Mykel J. Kochenderfer,et al.  Toward Scalable Verification for Safety-Critical Deep Networks , 2018, ArXiv.

[16]  Yasser Shoukry,et al.  Formal verification of neural network controlled autonomous systems , 2018, HSCC.

[17]  Nathan Fulton,et al.  Safe Reinforcement Learning via Formal Methods: Toward Safe Control Through Proof and Learning , 2018, AAAI.

[18]  Hasan Ferit Eniser,et al.  RAID: Randomized Adversarial-Input Detection for Neural Networks , 2020, ArXiv.

[19]  Srikanth Kandula,et al.  Resource Management with Deep Reinforcement Learning , 2016, HotNets.

[20]  Min Wu,et al.  Safety Verification of Deep Neural Networks , 2016, CAV.

[21]  Mykel J. Kochenderfer,et al.  Reluplex: a calculus for reasoning about deep neural networks , 2021, Formal Methods in System Design.

[22]  P. Cochat,et al.  Et al , 2008, Archives de pediatrie : organe officiel de la Societe francaise de pediatrie.

[23]  Kyle D. Julian,et al.  Parallelization Techniques for Verifying Neural Networks , 2020, 2020 Formal Methods in Computer Aided Design (FMCAD).

[24]  Justin Emile Gottschlich,et al.  An Abstraction-Based Framework for Neural Network Verification , 2019, CAV.

[25]  Guy Katz,et al.  Minimal Modifications of Deep Neural Networks using Verification , 2020, LPAR.

[26]  Neil Immerman,et al.  Decidability of inferring inductive invariants , 2016, POPL.

[27]  Yuxi Li,et al.  Deep Reinforcement Learning: An Overview , 2017, ArXiv.

[28]  Florian Lorber,et al.  Shield Synthesis for Reinforcement Learning , 2020, ISoLA.

[29]  Clark W. Barrett,et al.  Provably Minimally-Distorted Adversarial Examples , 2017 .

[30]  Mykel J. Kochenderfer,et al.  Global Optimization of Objective Functions Represented by ReLU Networks , 2021, Machine Learning.

[31]  Joan Bruna,et al.  Intriguing properties of neural networks , 2013, ICLR.

[32]  Dafna Shahaf,et al.  Learning To Route with Deep RL , 2017 .

[33]  L. D. Moura,et al.  The YICES SMT Solver , 2006 .

[34]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[35]  Mykel J. Kochenderfer,et al.  The Marabou Framework for Verification and Analysis of Deep Neural Networks , 2019, CAV.

[36]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[37]  Junfeng Yang,et al.  Formal Security Analysis of Neural Networks using Symbolic Intervals , 2018, USENIX Security Symposium.

[38]  Michael Schapira,et al.  Verifying Deep-RL-Driven Systems , 2019, NetAI@SIGCOMM.

[39]  Mykel J. Kochenderfer,et al.  Towards Proving the Adversarial Robustness of Deep Neural Networks , 2017, FVAV@iFM.

[40]  Thomas A. Henzinger,et al.  Handbook of Model Checking , 2018, Springer International Publishing.

[41]  Christel Baier,et al.  Principles of model checking , 2008 .

[42]  Clark Barrett,et al.  An SMT-Based Approach for Verifying Binarized Neural Networks , 2020, TACAS.

[43]  Cho-Jui Hsieh,et al.  Efficient Neural Network Robustness Certification with General Activation Functions , 2018, NeurIPS.

[44]  Xin Zhang,et al.  End to End Learning for Self-Driving Cars , 2016, ArXiv.

[45]  Jun Zhao,et al.  Recurrent Convolutional Neural Networks for Text Classification , 2015, AAAI.

[46]  Cesare Tinelli,et al.  Satisfiability Modulo Theories , 2021, Handbook of Satisfiability.

[47]  Edmund M. Clarke,et al.  Counterexample-guided abstraction refinement , 2003, 10th International Symposium on Temporal Representation and Reasoning, 2003 and Fourth International Conference on Temporal Logic. Proceedings..

[48]  Hongzi Mao,et al.  Neural Adaptive Video Streaming with Pensieve , 2017, SIGCOMM.

[49]  Alex Graves,et al.  Playing Atari with Deep Reinforcement Learning , 2013, ArXiv.

[50]  Brighten Godfrey,et al.  A Deep Reinforcement Learning Perspective on Internet Congestion Control , 2019, ICML.

[51]  Nikolaj Bjørner,et al.  Z3: An Efficient SMT Solver , 2008, TACAS.

[52]  Matthew Mirman,et al.  Fast and Effective Robustness Certification , 2018, NeurIPS.

[53]  F. Wolf,et al.  Static Neural Compiler Optimization via Deep Reinforcement Learning , 2020, 2020 IEEE/ACM 6th Workshop on the LLVM Compiler Infrastructure in HPC (LLVM-HPC) and Workshop on Hierarchical Parallelism for Exascale Computing (HiPar).

[54]  Corina S. Pasareanu,et al.  DeepSafe: A Data-Driven Approach for Assessing Robustness of Neural Networks , 2018, ATVA.

[55]  Luca Pulina,et al.  Challenging SMT solvers to verify neural networks , 2012, AI Commun..

[56]  Arnab Sarkar,et al.  Deep Learning based Prediction Model for Adaptive Video Streaming , 2020, 2020 International Conference on COMmunication Systems & NETworkS (COMSNETS).