SceneChecker: Boosting Scenario Verification Using Symmetry Abstractions

We present SceneChecker, a tool for verifying scenarios involving vehicles executing complex plans in large cluttered workspaces. SceneChecker converts the scenario verification problem to a standard hybrid system verification problem, and solves it effectively by exploiting structural properties in the plan and the vehicle dynamics. SceneChecker uses symmetry abstractions, a novel refinement algorithm, and importantly, is built to boost the performance of any existing reachability analysis tool as a plug-in subroutine. We evaluated SceneChecker on several scenarios involving ground and aerial vehicles with nonlinear dynamics and neural network controllers, employing different kinds of symmetries, using different reachability subroutines, and following plans with hundreds of waypoints in complex workspaces. Compared to two leading tools, DryVR and Flow*, SceneChecker shows 14× average speedup in verification time, even while using those very tools as reachability subroutines1.

[1]  Alberto L. Sangiovanni-Vincentelli,et al.  Scenic: a language for scenario specification and scene generation , 2018, PLDI.

[2]  A. Prasad Sistla,et al.  Symmetry and model checking , 1993, Formal Methods Syst. Des..

[3]  Germán Ros,et al.  CARLA: An Open Urban Driving Simulator , 2017, CoRL.

[4]  Stanley Bak,et al.  HyLAA: A Tool for Computing Simulation-Equivalent Reachability for Linear Systems , 2017, HSCC.

[5]  Makai Mann,et al.  Partial Order Reduction for Deep Bug Finding in Synchronous Hardware , 2020, TACAS.

[6]  Chuchu Fan,et al.  Fast and Guaranteed Safe Controller Synthesis for Nonlinear Vehicle Models , 2020, CAV.

[7]  Insup Lee,et al.  Verisig: verifying safety properties of hybrid systems with neural network controllers , 2018, HSCC.

[8]  Roderick Bloem,et al.  Parameterized Synthesis , 2012, TACAS.

[9]  Matthias Althoff,et al.  CommonRoad: Composable benchmarks for motion planning on roads , 2017, 2017 IEEE Intelligent Vehicles Symposium (IV).

[10]  John N. Maidens,et al.  Exploiting Symmetry for Discrete-Time Reachability Computations , 2018, IEEE Control Systems Letters.

[11]  Joost-Pieter Katoen,et al.  Symmetry reduction for stochastic hybrid systems , 2008, 2008 47th IEEE Conference on Decision and Control.

[12]  B. Faverjon,et al.  Probabilistic Roadmaps for Path Planning in High-Dimensional Con(cid:12)guration Spaces , 1996 .

[13]  Marco Caccamo,et al.  Safety and Progress for Distributed Cyber-Physical Systems with Unreliable Communication , 2015, ACM Trans. Embed. Comput. Syst..

[14]  Marta Z. Kwiatkowska,et al.  Symmetry Reduction for Probabilistic Model Checking , 2006, CAV.

[15]  Rupak Majumdar,et al.  Exploiting Symmetries to Speed Up SAT-Based Boolean Matching for Logic Synthesis of FPGAs , 2008, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems.

[16]  Kerstin Eder,et al.  Symmetry Reduction Enables Model Checking of More Complex Emergent Behaviours of Swarm Navigation Algorithms , 2015, TAROS.

[17]  Sayan Mitra,et al.  Symmetry Abstractions for Hybrid Systems and their Applications , 2020, ArXiv.

[18]  Russ Tedrake,et al.  Funnel libraries for real-time robust feedback motion planning , 2016, Int. J. Robotics Res..

[19]  Mahesh Viswanathan,et al.  Meeting a Powertrain Verification Challenge , 2015, CAV.

[20]  Sayan Mitra,et al.  Using Symmetry Transformations in Equivariant Dynamical Systems for Their Safety Verification , 2019, ATVA.

[21]  S. LaValle.  Rapidly-exploring random trees : a new tool for path planning , 1998 .

[22]  Xin Chen,et al.  Flow*: An Analyzer for Non-linear Hybrid Systems , 2013, CAV.

[23]  Mahesh Viswanathan,et al.  Automatic Reachability Analysis for Nonlinear Hybrid Models with C2E2 , 2016, CAV.

[24]  Tiziana Margaria,et al.  Tools and algorithms for the construction and analysis of systems: a special issue for TACAS 2017 , 2001, International Journal on Software Tools for Technology Transfer.

[25]  Antoine Girard,et al.  SpaceEx: Scalable Verification of Hybrid Systems , 2011, CAV.

[26]  J. Slotine,et al.  Symmetries, stability, and control in nonlinear systems and networks. , 2010, Physical review. E, Statistical, nonlinear, and soft matter physics.

[27]  Matthias Althoff,et al.  An Introduction to CORA 2015 , 2015, ARCH@CPSWeek.

[28]  Ashish Tiwari,et al.  Sherlock - A tool for verification of neural network feedback systems: demo abstract , 2019, HSCC.

[29]  Mahesh Viswanathan,et al.  DryVR: Data-Driven Verification and Compositional Reasoning for Automotive Systems , 2017, CAV.

[30]  Chuchu Fan,et al.  Multi-agent Safety Verification Using Symmetry Transformations , 2019, TACAS.

[31]  Weiming Xiang,et al.  NNV: The Neural Network Verification Tool for Deep Neural Networks and Learning-Enabled Cyber-Physical Systems , 2020, CAV.

[32]  David L. Dill,et al.  Better verification through symmetry , 1996, Formal Methods Syst. Des..

[33]  Lydia E. Kavraki,et al.  Probabilistic roadmaps for path planning in high-dimensional configuration spaces , 1996, IEEE Trans. Robotics Autom..

[34]  Somesh Jha,et al.  Symmetry and Induction in Model Checking , 1995, Computer Science Today.