Verification for Machine Learning, Autonomy, and Neural Networks Survey

This survey presents an overview of verification techniques for autonomous systems, with a focus on safety-critical autonomous cyber-physical systems (CPS) and subcomponents thereof. Autonomy in CPS is enabling by recent advances in artificial intelligence (AI) and machine learning (ML) through approaches such as deep neural networks (DNNs), embedded in so-called learning enabled components (LECs) that accomplish tasks from classification to control. Recently, the formal methods and formal verification community has developed methods to characterize behaviors in these LECs with eventual goals of formally verifying specifications for LECs, and this article presents a survey of many of these recent approaches.

[1]  Johann Schumann,et al.  A tool for verification and validation of neural network based adaptive controllers for high assurance systems , 2004, Eighth IEEE International Symposium on High Assurance Systems Engineering, 2004. Proceedings..

[2]  Robert M. Sanner,et al.  Gaussian Networks for Direct Adaptive Control , 1991, 1991 American Control Conference.

[3]  Amnon Shashua,et al.  Safe, Multi-Agent, Reinforcement Learning for Autonomous Driving , 2016, ArXiv.

[4]  Qingkai Liang,et al.  Accelerated Primal-Dual Policy Optimization for Safe Reinforcement Learning , 2018, ArXiv.

[5]  Jianbin Qiu,et al.  A Combined Adaptive Neural Network and Nonlinear Model Predictive Control for Multirate Networked Industrial Process Control , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[6]  Didrik Lundberg,et al.  Automated Inference of Excitable Cell Models as Hybrid Automata , 2013 .

[7]  Sameep Mehta,et al.  An Introduction to Adversarial Machine Learning , 2017, BDA.

[8]  Daniel Kroening,et al.  Concolic Testing for Deep Neural Networks , 2018, 2018 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE).

[9]  Pushmeet Kohli,et al.  A Dual Approach to Scalable Verification of Deep Networks , 2018, UAI.

[10]  Cees Witteveen,et al.  The efficiency of identifying timed automata and the power of clocks , 2011, Inf. Comput..

[11]  Anca D. Dragan,et al.  Pragmatic-Pedagogic Value Alignment , 2017, ISRR.

[12]  Ashish Tiwari,et al.  TeLEx: Passive STL Learning Using Only Positive Examples , 2017, RV.

[13]  Taylor T. Johnson,et al.  Periodically-Scheduled Controller Analysis Using Hybrid Systems Reachability and Continuization , 2015, 2015 IEEE Real-Time Systems Symposium.

[14]  Sanjit A. Seshia,et al.  Mining Requirements From Closed-Loop Control Models , 2015, IEEE Trans. Comput. Aided Des. Integr. Circuits Syst..

[15]  D. Ward,et al.  Verification and validation of neural networks for safety-critical applications , 2002, Proceedings of the 2002 American Control Conference (IEEE Cat. No.CH37301).

[16]  Peng Shi,et al.  Adaptive tracking control for switched stochastic nonlinear systems with unknown actuator dead-zone , 2015, Autom..

[17]  Leonid Ryzhyk,et al.  Verifying Properties of Binarized Deep Neural Networks , 2017, AAAI.

[18]  Kwang Y. Lee,et al.  Diagonal recurrent neural networks for dynamic systems control , 1995, IEEE Trans. Neural Networks.

[19]  Andreas Krause,et al.  Safe Model-based Reinforcement Learning with Stability Guarantees , 2017, NIPS.

[20]  Frank L. Lewis,et al.  Neural Networks in Feedback Control Systems , 2015 .

[21]  Weiming Xiang,et al.  Output Reachable Set Estimation and Verification for Multilayer Neural Networks , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[22]  Martin Fränzle,et al.  HySAT: An efficient proof engine for bounded model checking of hybrid systems , 2007, Formal Methods Syst. Des..

[23]  Benno Stein,et al.  Learning Behavior Models for Hybrid Timed Systems , 2012, AAAI.

[24]  Frank L. Lewis,et al.  Optimal Control: Lewis/Optimal Control 3e , 2012 .

[25]  Inderjit S. Dhillon,et al.  Towards Fast Computation of Certified Robustness for ReLU Networks , 2018, ICML.

[26]  Shuzhi Sam Ge,et al.  Neural Network Control of a Rehabilitation Robot by State and Output Feedback , 2015, J. Intell. Robotic Syst..

[27]  Lui Sha,et al.  Real-Time Reachability for Verified Simplex Design , 2014, RTSS.

[28]  Tim Kelly,et al.  Using fuzzy self-organising maps for safety critical systems , 2007, Reliab. Eng. Syst. Saf..

[29]  Daniel Liberzon,et al.  Calculus of Variations and Optimal Control Theory: A Concise Introduction , 2012 .

[30]  Wei He,et al.  Adaptive Neural Network Control of a Marine Vessel With Constraints Using the Asymmetric Barrier Lyapunov Function. , 2017, IEEE transactions on cybernetics.

[31]  Matthew Wicker,et al.  Feature-Guided Black-Box Safety Testing of Deep Neural Networks , 2017, TACAS.

[32]  Frits W. Vaandrager,et al.  Learning and Testing the Bounded Retransmission Protocol , 2012, ICGI.

[33]  Jaime F. Fisac,et al.  A General Safety Framework for Learning-Based Control in Uncertain Robotic Systems , 2017, IEEE Transactions on Automatic Control.

[34]  Weiming Xiang,et al.  Reachable Set Estimation and Safety Verification for Piecewise Linear Systems with Neural Network Controllers , 2018, 2018 Annual American Control Conference (ACC).

[35]  Rob Ashmore,et al.  The State of Solutions for Autonomous Systems Safety , 2018 .

[36]  Mo Chen,et al.  FaSTrack: A modular framework for fast and guaranteed safe motion planning , 2017, 2017 IEEE 56th Annual Conference on Decision and Control (CDC).

[37]  Frank L. Lewis,et al.  An approximate Dynamic Programming based controller for an underactuated 6DoF quadrotor , 2011, 2011 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL).

[38]  Cees Witteveen,et al.  Efficiently identifying deterministic real-time automata from labeled data , 2011, Machine Learning.

[39]  Warren E. Dixon,et al.  Approximate optimal trajectory tracking for continuous-time nonlinear systems , 2013, Autom..

[40]  Mykel J. Kochenderfer,et al.  Neural Network Guidance for UAVs , 2017 .

[41]  Junfeng Yang,et al.  Formal Security Analysis of Neural Networks using Symbolic Intervals , 2018, USENIX Security Symposium.

[42]  Antonio Criminisi,et al.  Measuring Neural Net Robustness with Constraints , 2016, NIPS.

[43]  Warren E. Dixon,et al.  Efficient model-based reinforcement learning for approximate online optimal control , 2015, Autom..

[44]  Shaocheng Tong,et al.  Neural Network Control-Based Adaptive Learning Design for Nonlinear Systems With Full-State Constraints , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[45]  He Zhang,et al.  Reachable set estimation for inertial Markov jump BAM neural network with partially unknown transition rates and bounded disturbances , 2017, J. Frankl. Inst..

[46]  Ufuk Topcu,et al.  Safe Reinforcement Learning via Shielding , 2017, AAAI.

[47]  R.R. Zakrzewski,et al.  Randomized approach to verification of neural networks , 2004, 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541).

[48]  Jascha Sohl-Dickstein,et al.  Sensitivity and Generalization in Neural Networks: an Empirical Study , 2018, ICLR.

[49]  Simon X. Yang,et al.  Observer-Based Adaptive Neural Network Trajectory Tracking Control for Remotely Operated Vehicle , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[50]  Jaime F. Fisac,et al.  Reachability-based safe learning with Gaussian processes , 2014, 53rd IEEE Conference on Decision and Control.

[51]  Weiming Xiang,et al.  Reachable Set Estimation and Verification for Neural Network Models of Nonlinear Dynamic Systems , 2018, Safe, Autonomous and Intelligent Vehicles.

[52]  Joan Bruna,et al.  Intriguing properties of neural networks , 2013, ICLR.

[53]  Shuzhi Sam Ge,et al.  Robust Adaptive Neural Network Control for a Class of Uncertain MIMO Nonlinear Systems With Input Nonlinearities , 2010, IEEE Transactions on Neural Networks.

[54]  Ashish Tiwari,et al.  Output Range Analysis for Deep Feedforward Neural Networks , 2018, NFM.

[55]  Suman Jana,et al.  DeepTest: Automated Testing of Deep-Neural-Network-Driven Autonomous Cars , 2017, 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE).

[56]  Karl Meinke,et al.  IDS: An Incremental Learning Algorithm for Finite Automata , 2012, ArXiv.

[57]  Mo Chen,et al.  Safe platooning of unmanned aerial vehicles via reachability , 2015, 2015 54th IEEE Conference on Decision and Control (CDC).

[58]  Harald Raffelt,et al.  LearnLib: a library for automata learning and experimentation , 2005, FMICS '05.

[59]  Daniel Kroening,et al.  Testing Deep Neural Networks , 2018, ArXiv.

[60]  Bin Xu,et al.  Robust adaptive neural control of flexible hypersonic flight vehicle with dead-zone input nonlinearity , 2015 .

[61]  Kumpati S. Narendra,et al.  Identification and control of dynamical systems using neural networks , 1990, IEEE Trans. Neural Networks.

[62]  Cong Wang,et al.  Dynamic Learning From Adaptive Neural Network Control of a Class of Nonaffine Nonlinear Systems , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[63]  Shuzhi Sam Ge,et al.  Robust adaptive control of a thruster assisted position mooring system , 2014, Autom..

[64]  Somesh Jha,et al.  Semantic Adversarial Deep Learning , 2018, IEEE Design & Test.

[65]  Jinfeng Yi,et al.  Evaluating the Robustness of Neural Networks: An Extreme Value Theory Approach , 2018, ICLR.

[66]  Xiaoping Liu,et al.  Robust Adaptive Neural Tracking Control for a Class of Stochastic Nonlinear Interconnected Systems , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[67]  Frits W. Vaandrager,et al.  Applying Automata Learning to Embedded Control Software , 2015, ICFEM.

[68]  Philip S. Thomas,et al.  Safe Reinforcement Learning , 2015 .

[69]  Toshio Fukuda,et al.  Theory and applications of neural networks for industrial control systems , 1992, IEEE Trans. Ind. Electron..

[70]  Claire J. Tomlin,et al.  Guaranteed Safe Online Learning via Reachability: tracking a ground target using a quadrotor , 2012, 2012 IEEE International Conference on Robotics and Automation.

[71]  Dejan Nickovic,et al.  Parametric Identification of Temporal Properties , 2011, RV.

[72]  Rachid Guerraoui,et al.  Virtuously Safe Reinforcement Learning , 2018, ArXiv.

[73]  Burr Settles,et al.  Active Learning Literature Survey , 2009 .

[74]  Javier García,et al.  A comprehensive survey on safe reinforcement learning , 2015, J. Mach. Learn. Res..

[75]  Tiziana Margaria,et al.  Next Generation LearnLib , 2011, TACAS.

[76]  Matthias Hein,et al.  Formal Guarantees on the Robustness of a Classifier against Adversarial Manipulation , 2017, NIPS.

[77]  Sanjit A. Seshia,et al.  Compositional Falsification of Cyber-Physical Systems with Machine Learning Components , 2017, NFM.

[78]  Michael Mateas,et al.  CHARDA: Causal Hybrid Automata Recovery via Dynamic Analysis , 2017, IJCAI.

[79]  Kazuo Tanaka,et al.  An approach to stability criteria of neural-network control systems , 1996, IEEE Trans. Neural Networks.

[80]  Shuang Cong,et al.  PID-Like Neural Network Nonlinear Adaptive Control for Uncertain Multivariable Motion Control Systems , 2009, IEEE Transactions on Industrial Electronics.

[81]  Alessio Lomuscio,et al.  An approach to reachability analysis for feed-forward ReLU neural networks , 2017, ArXiv.

[82]  Mykel J. Kochenderfer,et al.  Policy compression for aircraft collision avoidance systems , 2016, 2016 IEEE/AIAA 35th Digital Avionics Systems Conference (DASC).

[83]  Shuzhi Sam Ge,et al.  Adaptive Control of a Flexible Crane System With the Boundary Output Constraint , 2014, IEEE Transactions on Industrial Electronics.

[84]  Huaguang Zhang,et al.  Neural-Network-Based Constrained Optimal Control Scheme for Discrete-Time Switched Nonlinear System Using Dual Heuristic Programming , 2014, IEEE Transactions on Automation Science and Engineering.

[85]  John C. Duchi,et al.  Certifying Some Distributional Robustness with Principled Adversarial Training , 2017, ICLR.

[86]  Russ Tedrake,et al.  Verifying Neural Networks with Mixed Integer Programming , 2017, ArXiv.

[87]  Rushikesh Kamalapurkar,et al.  Concurrent learning-based approximate optimal regulation , 2013, 52nd IEEE Conference on Decision and Control.

[88]  Xiaowei Huang,et al.  Reachability Analysis of Deep Neural Networks with Provable Guarantees , 2018, IJCAI.

[89]  Zhongke Shi,et al.  Composite Neural Dynamic Surface Control of a Class of Uncertain Nonlinear Systems in Strict-Feedback Form , 2014, IEEE Transactions on Cybernetics.

[90]  Luca Pulina,et al.  Automated Verification of Neural Networks: Advances, Challenges and Perspectives , 2018, ArXiv.

[91]  Anil A. Bharath,et al.  Deep Reinforcement Learning: A Brief Survey , 2017, IEEE Signal Processing Magazine.

[92]  Derong Liu,et al.  Neural-network-based adaptive optimal tracking control scheme for discrete-time nonlinear systems with approximation errors , 2013, Neurocomputing.

[93]  Snehasis Mukhopadhyay,et al.  Adaptive control using neural networks and approximate models , 1997, IEEE Trans. Neural Networks.

[94]  Luca Pulina,et al.  NeVer: a tool for artificial neural networks verification , 2011, Annals of Mathematics and Artificial Intelligence.

[95]  Pat Langley,et al.  The changing science of machine learning , 2011, Machine Learning.

[96]  James Kapinski,et al.  INVITED: Reasoning about Safety of Learning-Enabled Components in Autonomous Cyber-physical Systems , 2018, 2018 55th ACM/ESDA/IEEE Design Automation Conference (DAC).

[97]  Warren E. Dixon,et al.  Reinforcement Learning for Optimal Feedback Control , 2018 .

[98]  Georgios Fainekos,et al.  Simulation-based Adversarial Test Generation for Autonomous Vehicles with Machine Learning Components , 2018, 2018 IEEE Intelligent Vehicles Symposium (IV).

[99]  Marc G. Bellemare,et al.  Safe and Efficient Off-Policy Reinforcement Learning , 2016, NIPS.

[100]  Paulo Tabuada,et al.  Underminer , 2017, ACM Trans. Embed. Comput. Syst..

[101]  Pushmeet Kohli,et al.  Piecewise Linear Neural Network verification: A comparative study , 2017, ArXiv.

[102]  Sebastian Fischmeister,et al.  A framework for mining hybrid automata from input/output traces , 2015, 2015 International Conference on Embedded Software (EMSOFT).

[103]  Sebastian Fischmeister,et al.  Periodic Task Mining in Embedded System Traces , 2017, 2017 IEEE Real-Time and Embedded Technology and Applications Symposium (RTAS).

[104]  Luca Pulina,et al.  Challenging SMT solvers to verify neural networks , 2012, AI Commun..

[105]  Cees Witteveen,et al.  An algorithm for learning real-time automata , 2007 .

[106]  Lei Ma,et al.  DeepGauge: Comprehensive and Multi-Granularity Testing Criteria for Gauging the Robustness of Deep Learning Systems , 2018, ArXiv.

[107]  Xiaolin Hu,et al.  Discovering Adversarial Examples with Momentum , 2017, ArXiv.

[108]  Huaguang Zhang,et al.  Observer-Based Adaptive Neural Network Control for Nonlinear Systems in Nonstrict-Feedback Form , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[109]  I. V. Ramakrishnan,et al.  Learning Cycle-Linear Hybrid Automata for Excitable Cells , 2007, HSCC.

[110]  Susmit Jha,et al.  On Learning Sparse Boolean Formulae for Explaining AI Decisions , 2017, NFM.

[111]  B. Bavarian,et al.  Introduction to neural networks for intelligent control , 1988, IEEE Control Systems Magazine.

[112]  Owain Evans,et al.  Trial without Error: Towards Safe Reinforcement Learning via Human Intervention , 2017, AAMAS.

[113]  Daniel Kroening,et al.  Global Robustness Evaluation of Deep Neural Networks with Provable Guarantees for L0 Norm , 2018, ArXiv.

[114]  Chih-Hong Cheng,et al.  Maximum Resilience of Artificial Neural Networks , 2017, ATVA.

[115]  Robert J. Vanderbei,et al.  Linear Programming: Foundations and Extensions , 1998, Kluwer international series in operations research and management service.

[116]  Andrew W. Moore,et al.  Reinforcement Learning: A Survey , 1996, J. Artif. Intell. Res..

[117]  Changyin Sun,et al.  Adaptive Neural Impedance Control of a Robotic Manipulator With Input Saturation , 2016, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[118]  Mohamed Chtourou,et al.  A New PID Neural Network Controller Design for Nonlinear Processes , 2018, J. Circuits Syst. Comput..

[119]  Mykel J. Kochenderfer,et al.  Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks , 2017, CAV.

[120]  Frits W. Vaandrager,et al.  Combining Model Learning and Model Checking to Analyze TCP Implementations , 2016, CAV.

[121]  Sergiy Bogomolov,et al.  PDDL+ Planning with Hybrid Automata: Foundations of Translating Must Behavior , 2015, ICAPS.

[122]  Sicco Verwer.  Efficient Identification of Timed Automata: Theory and practice , 2010 .

[123]  Quanmin Zhu,et al.  Stable adaptive neurocontrol for nonlinear discrete-time systems , 2004, IEEE Trans. Neural Networks.

[124]  Stavros Tripakis,et al.  Learning Moore machines from input–output traces , 2016, International Journal on Software Tools for Technology Transfer.

[125]  Sen Wang,et al.  Deep Reinforcement Learning for Autonomous Driving , 2018, ArXiv.

[126]  Swarat Chaudhuri,et al.  AI2: Safety and Robustness Certification of Neural Networks with Abstract Interpretation , 2018, 2018 IEEE Symposium on Security and Privacy (SP).

[127]  Frank L. Lewis,et al.  2009 Special Issue: Neural network approach to continuous-time direct adaptive optimal control for partially unknown nonlinear systems , 2009 .

[128]  Dana Angluin,et al.  Queries and concept learning , 1988, Machine Learning.

[129]  Alexandre Donzé,et al.  Breach, A Toolbox for Verification and Parameter Synthesis of Hybrid Systems , 2010, CAV.

[130]  Dimitri P. Bertsekas,et al.  Dynamic Programming and Optimal Control, Two Volume Set , 1995 .

[131]  Xin Zhang,et al.  End to End Learning for Self-Driving Cars , 2016, ArXiv.

[132]  Guo-Xing Wen,et al.  Fuzzy Neural Network-Based Adaptive Control for a Class of Uncertain Nonlinear Stochastic Systems , 2014, IEEE Transactions on Cybernetics.

[133]  J. Zico Kolter,et al.  Provable defenses against adversarial examples via the convex outer adversarial polytope , 2017, ICML.

[134]  Bernd Becker,et al.  Towards Verification of Artificial Neural Networks , 2015, MBMV.

[135]  Lei Ma,et al.  DeepHunter: Hunting Deep Neural Network Defects via Coverage-Guided Fuzzing , 2018, 1809.01266.

[136]  Mo Chen,et al.  Reachability-Based Safety and Goal Satisfaction of Unmanned Aerial Platoons on Air Highways , 2016, 1602.08150.

[137]  Nathan Fulton,et al.  Safe Reinforcement Learning via Formal Methods: Toward Safe Control Through Proof and Learning , 2018, AAAI.

[138]  Manfred Morari,et al.  Multi-Parametric Toolbox 3.0 , 2013, 2013 European Control Conference (ECC).

[139]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.

[140]  Luca Pulina,et al.  An Abstraction-Refinement Approach to Verification of Artificial Neural Networks , 2010, CAV.

[141]  Weiming Xiang,et al.  Reachable Set Computation and Safety Verification for Neural Networks with ReLU Activations , 2017, ArXiv.

[142]  David K. Smith,et al.  Dynamic Programming and Optimal Control. Volume 1 , 1996 .

[143]  Jianxiong Xiao,et al.  DeepDriving: Learning Affordance for Direct Perception in Autonomous Driving , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[144]  Chih-Hong Cheng,et al.  Verification of Binarized Neural Networks , 2017, ArXiv.

[145]  Sicco Verwer,et al.  Car-following Behavior Model Learning Using Timed Automata , 2017 .

[146]  Junfeng Yang,et al.  DeepXplore: Automated Whitebox Testing of Deep Learning Systems , 2017, SOSP.

[147]  Jin Zhang,et al.  Neural-network control of nonaffine nonlinear system with zero dynamics by state and output feedback , 2003, IEEE Trans. Neural Networks.

[148]  Luca Pulina,et al.  Verification and repair of control policies for safe reinforcement learning , 2017, Applied Intelligence.

[149]  Fredrik Olsson,et al.  A literature survey of active machine learning in the context of natural language processing , 2009 .

[150]  Peng Shi,et al.  Intelligent Tracking Control for a Class of Uncertain High-Order Nonlinear Systems , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[151]  Bing Chen,et al.  Neural-Based Adaptive Output-Feedback Control for a Class of Nonstrict-Feedback Stochastic Nonlinear Systems , 2015, IEEE Transactions on Cybernetics.

[152]  Avimanyu Sahoo,et al.  Neural Network-Based Event-Triggered State Feedback Control of Nonlinear Continuous-Time Systems , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[153]  Aditi Raghunathan,et al.  Certified Defenses against Adversarial Examples , 2018, ICLR.

[154]  Sumit Gulwani,et al.  Neural-Guided Deductive Search for Real-Time Program Synthesis from Examples , 2018, ICLR.

[155]  Bernhard Steffen,et al.  The Open-Source LearnLib - A Framework for Active Automata Learning , 2015, CAV.

[156]  Frank L. Lewis,et al.  Online actor critic algorithm to solve the continuous-time infinite horizon optimal control problem , 2009, 2009 International Joint Conference on Neural Networks.

[157]  Andrew G. Barto,et al.  Lyapunov Design for Safe Reinforcement Learning , 2003, J. Mach. Learn. Res..

[158]  Ashish Tiwari,et al.  Output Range Analysis for Deep Neural Networks , 2017, ArXiv.

[159]  Rüdiger Ehlers,et al.  Formal Verification of Piece-Wise Linear Feed-Forward Neural Networks , 2017, ATVA.