Compositional Falsification of Cyber-Physical Systems with Machine Learning Components

Cyber-physical systems (CPS), such as automotive systems, are starting to include sophisticated machine learning (ML) components. Their correctness, therefore, depends on properties of the inner ML modules. While learning algorithms aim to generalize from examples, they are only as good as the examples provided, and recent efforts have shown that they can produce inconsistent output under small adversarial perturbations. This raises the question: can the output from learning components can lead to a failure of the entire CPS? In this work, we address this question by formulating it as a problem of falsifying signal temporal logic (STL) specifications for CPS with ML components. We propose a compositional falsification framework where a temporal logic falsifier and a machine learning analyzer cooperate with the aim of finding falsifying executions of the considered model. The efficacy of the proposed technique is shown on an automatic emergency braking system model with a perception component based on deep neural networks.

[1]  H. Weyl.  Über die Gleichverteilung von Zahlen mod. Eins , 1916 .

[2]  John R. Anderson,et al.  MACHINE LEARNING An Artificial Intelligence Approach , 2009 .

[3]  H. Niederreiter.  Low-discrepancy and low-dispersion sequences , 1988 .

[4]  Vladimir Vapnik,et al.  Principles of Risk Minimization for Learning Theory , 1991, NIPS.

[5]  Peter Shirley,et al.  Discrepancy as a Quality Measure for Sample Distributions , 1991, Eurographics.

[6]  Harald Niederreiter,et al.  Random number generation and Quasi-Monte Carlo methods , 1992, CBMS-NSF regional conference series in applied mathematics.

[7]  Blake Hannaford,et al.  Resolution-First Scanning of Multidimensional Spaces , 1993, CVGIP Graph. Model. Image Process..

[8]  I. Sloan.  Lattice Methods for Multiple Integration , 1994 .

[9]  Russel E. Caflisch,et al.  Quasi-Random Sequences and Their Discrepancies , 1994, SIAM J. Sci. Comput..

[10]  J. Rosenblatt,et al.  Ergodic Theory and its Connections with Harmonic Analysis: Pointwise ergodic theorems via harmonic analysis , 1995 .

[11]  Pat Langley,et al.  Selection of Relevant Features and Examples in Machine Learning , 1997, Artif. Intell..

[12]  Steven M. LaValle,et al.  Quasi-randomized path planning , 2001, Proceedings 2001 ICRA. IEEE International Conference on Robotics and Automation (Cat. No.01CH37164).

[13]  Dejan Nickovic,et al.  Monitoring Temporal Properties of Continuous Signals , 2004, FORMATS/FTRTFT.

[14]  J. Matousek,et al.  Geometric Discrepancy: An Illustrated Guide , 2009 .

[15]  Alexandre Donzé,et al.  Breach, A Toolbox for Verification and Parameter Synthesis of Hybrid Systems , 2010, CAV.

[16]  Jaewan Lee,et al.  Development and Evaluations of Advanced Emergency Braking System Algorithm for the Commercial Vehicle , 2011 .

[17]  Sriram Sankaranarayanan,et al.  S-TaLiRo: A Tool for Temporal Logic Falsification for Hybrid Systems , 2011, TACAS.

[18]  P. Cochat,et al.  Et al , 2008, Archives de pediatrie : organe officiel de la Societe francaise de pediatrie.

[19]  Tara N. Sainath,et al.  Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups , 2012, IEEE Signal Processing Magazine.

[20]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[21]  Thomas Ferrère,et al.  Efficient Robust Monitoring for STL , 2013, CAV.

[22]  Trevor Darrell,et al.  Caffe: Convolutional Architecture for Fast Feature Embedding , 2014, ACM Multimedia.

[23]  Joan Bruna,et al.  Intriguing properties of neural networks , 2013, ICLR.

[24]  Jason Yosinski,et al.  Deep neural networks are easily fooled: High confidence predictions for unrecognizable images , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Mahesh Viswanathan,et al.  C2E2: A Verification Tool for Stateflow Models , 2015, TACAS.

[26]  James Kapinski,et al.  Efficient Guiding Strategies for Testing of Temporal Properties of Hybrid Systems , 2015, NFM.

[27]  Xin Zhang,et al.  End to End Learning for Self-Driving Cars , 2016, ArXiv.

[28]  Seyed-Mohsen Moosavi-Dezfooli,et al.  DeepFool: A Simple and Accurate Method to Fool Deep Neural Networks , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  Martín Abadi,et al.  TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems , 2016, ArXiv.

[30]  Sanjit A. Seshia,et al.  Towards Verified Artificial Intelligence , 2016, ArXiv.

[31]  Sanjit A. Seshia,et al.  Combining requirement mining, software model checking and simulation-based verification for industrial automotive systems , 2016, 2016 Formal Methods in Computer-Aided Design (FMCAD).

[32]  Alberto L. Sangiovanni-Vincentelli,et al.  Systematic Testing of Convolutional Neural Networks for Autonomous Driving , 2017, ArXiv.

[33]  Junfeng Yang,et al.  DeepXplore: Automated Whitebox Testing of Deep Learning Systems , 2017, SOSP.

[34]  Mykel J. Kochenderfer,et al.  Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks , 2017, CAV.

[35]  Min Wu,et al.  Safety Verification of Deep Neural Networks , 2016, CAV.

[36]  Sanjit A. Seshia,et al.  Logical Clustering and Learning for Time-Series Data , 2016, 1612.07823.

[37]  David A. Wagner,et al.  Towards Evaluating the Robustness of Neural Networks , 2016, 2017 IEEE Symposium on Security and Privacy (SP).

[38]  Pascal Frossard,et al.  Analysis of classifiers’ robustness to adversarial perturbations , 2015, Machine Learning.

[39]  Somesh Jha,et al.  Semantic Adversarial Deep Learning , 2018, IEEE Design & Test.

[40]  Sanjit A. Seshia,et al.  Compositional Falsification of Cyber-Physical Systems with Machine Learning Components , 2017, Journal of Automated Reasoning.

[41]  Sanjit A. Seshia,et al.  Formal Specification for Deep Neural Networks , 2018, ATVA.