Evaluation Uncertainty in Data-Driven Self-Driving Testing

Safety evaluation of self-driving technologies has been extensively studied. One recent approach uses Monte Carlo based evaluation to estimate the occurrence probabilities of safety-critical events as safety measures. These Monte Carlo samples are generated from stochastic input models constructed based on real-world data. In this paper, we propose an approach to assess the impact on the probability estimates from the evaluation procedures due to the estimation error caused by data variability. Our proposed method merges the classical bootstrap method for estimating input uncertainty with a likelihood ratio based scheme to reuse experiment outputs. This approach is economical and efficient in terms of implementation costs in assessing input uncertainty for the evaluation of self-driving technology. We use an example in autonomous vehicle (AV) safety evaluation to demonstrate the proposed approach as a diagnostic tool for the quality of the fitted input model.

[1]  Stephen E. Chick,et al.  Bayesian Ideas and Discrete Event Simulation: Why, What and How , 2006, Proceedings of the 2006 Winter Simulation Conference.

[2]  David Hinkley,et al.  Bootstrap Methods: Another Look at the Jackknife , 2008 .

[3]  Shane G. Henderson.  Input model uncertainty: why do we care and what should we do about it? , 2003, Proceedings of the 2003 Winter Simulation Conference, 2003..

[4]  Peter W. Glynn,et al.  Stochastic Simulation: Algorithms and Analysis , 2007 .

[5]  P. Frazier,et al.  Advanced tutorial: Input uncertainty and robust analysis in stochastic simulation , 2016, 2016 Winter Simulation Conference (WSC).

[6]  Wei Xie,et al.  A framework for input uncertainty analysis , 2010, Proceedings of the 2010 Winter Simulation Conference.

[7]  Ding Zhao,et al.  Accelerated Evaluation of Automated Vehicles in Car-Following Maneuvers , 2016, IEEE Transactions on Intelligent Transportation Systems.

[8]  Nidhi Kalra,et al.  Driving to Safety , 2016 .

[9]  T. Kanade,et al.  Monte Carlo road safety reasoning , 2005, IEEE Proceedings. Intelligent Vehicles Symposium, 2005..

[10]  Russell R. Barton.  Input uncertainty in outout analysis , 2012, Winter Simulation Conference.

[11]  Ding Zhao,et al.  Accelerated Evaluation of Automated Vehicles. , 2016 .

[12]  Ding Zhao,et al.  Accelerated Evaluation of Automated Vehicles Using Piecewise Mixture Models , 2017, IEEE Transactions on Intelligent Transportation Systems.

[13]  B. Efron.  The jackknife, the bootstrap, and other resampling plans , 1987 .

[14]  R. Cheng,et al.  Sensitivity of computer simulation experiments to errors in input data , 1997 .

[15]  Barry L. Nelson,et al.  Foundations and Methods of Stochastic Simulation: A First Course , 2013 .

[16]  Lars Petersson,et al.  Statistical Threat Assessment for General Road Scenes Using Monte Carlo Sampling , 2008, IEEE Transactions on Intelligent Transportation Systems.

[17]  Barry L. Nelson,et al.  Advanced tutorial: Input uncertainty quantification , 2014, Proceedings of the Winter Simulation Conference 2014.