Reachability Analysis for Neural Agent-Environment Systems

We develop a novel model for studying agent-environment systems, where the agents are implemented via feed-forward ReLU neural networks. We provide a semantics and develop a method to verify automatically that no unwanted states are reached by the system during its evolution. We study several reachability problems for the system, ranging from one-step reachability, to fixed multi-step and arbitrary-step to study the system evolution. We also study the decision problem of whether an agent, realised via feed-forward ReLU networks will perform an action in a system run. Whenever possible, we give tight complexity bounds to decision problems introduced. We automate the various reachability problems studied by recasting them as mixed-integer linear programming problems. We present an implementation and discuss the experimental results obtained on a range of test cases.

[1]  Benjamin Schrauwen,et al.  Deep content-based music recommendation , 2013, NIPS.

[2]  Pushmeet Kohli,et al.  Piecewise Linear Neural Network verification: A comparative study , 2017, ArXiv.

[3]  Stuart J. Russell,et al.  Research Priorities for Robust and Beneficial Artificial Intelligence , 2015, AI Mag..

[4]  Chih-Hong Cheng,et al.  Maximum Resilience of Artificial Neural Networks , 2017, ATVA.

[5]  Quoc V. Le,et al.  Sequence to Sequence Learning with Neural Networks , 2014, NIPS.

[6]  Simon Haykin,et al.  Neural Networks and Learning Machines , 2010 .

[7]  Alessio Lomuscio,et al.  An approach to reachability analysis for feed-forward ReLU neural networks , 2017, ArXiv.

[8]  Wayne L. Winston.  Operations research: applications and algorithms / Wayne L. Winston , 2004 .

[9]  Donald Estep,et al.  Piecewise Linear Approximation , 2004 .

[10]  Alessio Lomuscio,et al.  Verification of Multi-Agent Systems via Predicate Abstraction against ATLK Specifications , 2016, AAMAS.

[11]  Rüdiger Ehlers,et al.  Formal Verification of Piece-Wise Linear Feed-Forward Neural Networks , 2017, ATVA.

[12]  Andreas Zell,et al.  SNNS (Stuttgart Neural Network Simulator) , 1994 .

[13]  Min Wu,et al.  Safety Verification of Deep Neural Networks , 2016, CAV.

[14]  Antonio Criminisi,et al.  Measuring Neural Net Robustness with Constraints , 2016, NIPS.

[15]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[16]  E. L. Lawler,et al.  Branch-and-Bound Methods: A Survey , 1966, Oper. Res..

[17]  Tim Kelly,et al.  Establishing Safety Criteria for Artificial Neural Networks , 2003, KES.

[18]  Stefan Korecko,et al.  Reachability Analysis of Time-Critical Systems , 2010 .

[19]  George B. Dantzig,et al.  Linear programming and extensions , 1965 .

[20]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .

[21]  Alessio Lomuscio,et al.  Verifying Emergent Properties of Swarms , 2015, IJCAI.

[22]  Fred Kröger,et al.  Temporal Logic of Programs , 1987, EATCS Monographs on Theoretical Computer Science.

[23]  Javier Esparza,et al.  Reachability Analysis of Multithreaded Software with Asynchronous Communication , 2005, FSTTCS.

[24]  Barbara Messing,et al.  An Introduction to MultiAgent Systems , 2002, Künstliche Intell..

[25]  Hector J. Levesque,et al.  The cognitive agents specification language and verification environment for multiagent systems , 2002, AAMAS '02.

[26]  Mehdi Dastani,et al.  Using Theorem Proving to Verify Properties of Agent Programs , 2010 .

[27]  Ron van der Meyden,et al.  MCK: Model Checking the Logic of Knowledge , 2004, CAV.

[28]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.

[29]  Wojciech Zaremba,et al.  OpenAI Gym , 2016, ArXiv.

[30]  Geoffrey E. Hinton,et al.  Rectified Linear Units Improve Restricted Boltzmann Machines , 2010, ICML.

[31]  Mykel J. Kochenderfer,et al.  Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks , 2017, CAV.

[32]  Alessio Lomuscio,et al.  MCMAS: A Model Checker for the Verification of Multi-Agent Systems , 2009, CAV.