Scalable End-to-End Autonomous Vehicle Testing via Rare-event Simulation

While recent developments in autonomous vehicle (AV) technology highlight substantial progress, we lack tools for rigorous and scalable testing. Real-world testing, the $\textit{de facto}$ evaluation environment, places the public in danger, and, due to the rare nature of accidents, will require billions of miles in order to statistically validate performance claims. We implement a simulation framework that can test an entire modern autonomous driving system, including, in particular, systems that employ deep-learning perception and control algorithms. Using adaptive importance-sampling methods to accelerate rare-event probability evaluation, we estimate the probability of an accident under a base distribution governing standard traffic behavior. We demonstrate our framework on a highway scenario, accelerating system evaluation by $2$-$20$ times over naive Monte Carlo sampling methods and $10$-$300 \mathsf{P}$ times (where $\mathsf{P}$ is the number of processors) over real-world testing.

[1]  P. Varaiya,et al.  What ' s Decidable about Hybrid Automata ? 1 , 1995 .

[2]  M. R. Rao,et al.  Combinatorial Optimization , 1992, NATO ASI Series.

[3]  Pravin Varaiya,et al.  What's decidable about hybrid automata? , 1995, STOC '95.

[4]  이동욱 12. M & A , 2000 .

[5]  R. Rubinstein Combinatorial Optimization, Cross-Entropy, Ants and Rare Events , 2001 .

[6]  Katja Vogel,et al.  A comparison of headway and time to collision as safety indicators. , 2003, Accident; analysis and prevention.

[7]  Zelda B. Zabinsky Annealing Adaptive Search , 2003 .

[8]  Zelda B. Zabinsky,et al.  Stochastic Adaptive Search for Global Optimization , 2003 .

[9]  John Lygeros,et al.  Lecture Notes on Hybrid Systems , 2004 .

[10]  Harald Heinecke,et al.  AUTomotive Open System ARchitecture - An Industry-Wide Initiative to Manage the Complexity of Emerging Automotive E/E-Architectures , 2004 .

[11]  Peter W. Glynn,et al.  Stochastic Simulation: Algorithms and Analysis , 2007 .

[12]  Tito Homem-de-Mello,et al.  A Study on the Cross-Entropy Method for Rare-Event Probability Estimation , 2007, INFORMS J. Comput..

[13]  R. Tibshirani,et al.  Sparse inverse covariance estimation with the graphical lasso. , 2008, Biostatistics.

[14]  Christel Baier,et al.  Principles of model checking , 2008 .

[15]  Ping Hu,et al.  On the performance of the Cross-Entropy method , 2009, Proceedings of the 2009 Winter Simulation Conference (WSC).

[16]  Gerardo Rubino,et al.  Introduction to Rare Event Simulation , 2009, Rare Event Simulation using Monte Carlo Methods.

[17]  J. Andrew Bagnell,et al.  Efficient Reductions for Imitation Learning , 2010, AISTATS.

[18]  Alex Graves,et al.  Practical Variational Inference for Neural Networks , 2011, NIPS.

[19]  Jiaqiao Hu,et al.  Annealing adaptive search, cross‐entropy, and stochastic approximation in global optimization , 2011 .

[20]  Geoffrey J. Gordon,et al.  A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning , 2010, AISTATS.

[21]  Marta Z. Kwiatkowska,et al.  PRISM 4.0: Verification of Probabilistic Real-Time Systems , 2011, CAV.

[22]  P. Cochat,et al.  Et al , 2008, Archives de pediatrie : organe officiel de la Societe francaise de pediatrie.

[23]  Ping Hu,et al.  A Stochastic Approximation Framework for a Class of Randomized Optimization Algorithms , 2012, IEEE Transactions on Automatic Control.

[24]  Pieter Hintjens,et al.  ZeroMQ: Messaging for Many Applications , 2013 .

[25]  Dirk P. Kroese,et al.  The cross-entropy method for estimation , 2013 .

[26]  Michiel van Ratingen,et al.  Implementation of Autonomous Emergency Braking (AEB), the Next Step in Euro NCAP's Safety Assessment , 2013 .

[27]  Aditya Bhaskara,et al.  Provable Bounds for Learning Some Deep Representations , 2013, ICML.

[28]  Jiaqiao Hu,et al.  Gradient-Based Adaptive Stochastic Search for Non-Differentiable Optimization , 2013, IEEE Transactions on Automatic Control.

[29]  Jiaqiao Hu,et al.  Model-Based Annealing Random Search with Stochastic Averaging , 2014, TOMC.

[30]  Matthias Althoff,et al.  Online Verification of Automated Road Vehicles Using Reachability Analysis , 2014, IEEE Transactions on Robotics.

[31]  Nadav Cohen,et al.  On the Expressive Power of Deep Learning: A Tensor Analysis , 2015, COLT 2016.

[32]  S. Shankar Sastry,et al.  Formal methods for semi-autonomous driving , 2015, 2015 52nd ACM/EDAC/IEEE Design Automation Conference (DAC).

[33]  Julien Cornebise,et al.  Weight Uncertainty in Neural Networks , 2015, ArXiv.

[34]  Ariel D. Procaccia,et al.  Variational Dropout and the Local Reparameterization Trick , 2015, NIPS.

[35]  Xin Zhang,et al.  End to End Learning for Self-Driving Cars , 2016, ArXiv.

[36]  Shinpei Kato,et al.  APEX: Autonomous Vehicle Plan Verification and Execution , 2016 .

[37]  Stefano Ermon,et al.  Generative Adversarial Imitation Learning , 2016, NIPS.

[38]  Benjamin Van Roy,et al.  Deep Exploration via Bootstrapped DQN , 2016, NIPS.

[39]  Georgios E. Fainekos,et al.  Utilizing S-TaLiRo as an automatic test generation framework for autonomous vehicles , 2016, 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC).

[40]  Ding Zhao,et al.  Accelerated evaluation of automated vehicles using extracted naturalistic driving data , 2016 .

[41]  Iyad Rahwan,et al.  The social dilemma of autonomous vehicles , 2015, Science.

[42]  Nidhi Kalra,et al.  Driving to Safety , 2016 .

[43]  Nguyen Hoang Nga,et al.  Combining Third Party Components Securely in Automotive Systems , 2016, WISTP.

[44]  Zoubin Ghahramani,et al.  Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning , 2015, ICML.

[45]  Germán Ros,et al.  CARLA: An Open Urban Driving Simulator , 2017, CoRL.

[46]  Shie Mannor,et al.  End-to-End Differentiable Adversarial Imitation Learning , 2017, ICML.

[47]  Amnon Shashua,et al.  On a Formal Model of Safe and Scalable Self-driving Cars , 2017, ArXiv.

[48]  Matus Telgarsky,et al.  Spectrally-normalized margin bounds for neural networks , 2017, NIPS.

[49]  Ashish Kapoor,et al.  AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles , 2017, FSR.

[50]  Mykel J. Kochenderfer,et al.  Imitating driver behavior with generative adversarial networks , 2017, 2017 IEEE Intelligent Vehicles Symposium (IV).

[51]  Mykel J. Kochenderfer,et al.  Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks , 2017, CAV.

[52]  Houssam Abbas,et al.  Safe At Any Speed: A Simulation-Based Test Harness for Autonomous Vehicles , 2017, CyPhy.

[53]  Min Wu,et al.  Safety Verification of Deep Neural Networks , 2016, CAV.

[54]  Russ Tedrake,et al.  Verifying Neural Networks with Mixed Integer Programming , 2017, ArXiv.

[55]  Insup Lee,et al.  Self-Driving Vehicle Verification Towards a Benchmark , 2018, ArXiv.

[56]  Ding Zhao,et al.  Accelerated Evaluation of Automated Vehicles in Car-Following Maneuvers , 2016, IEEE Transactions on Intelligent Transportation Systems.

[57]  Tsuyoshi Murata,et al.  {m , 1934, ACML.