How to Evaluate Self-Driving Testing Ground? A Quantitative Approach

Testing ground has been a critical component in testing and validation for Connected and Automated Vehicles, or CAV. Although quite a few world-class testing facilities have been under construction over the years, the evaluation of testing grounds themselves as testing approaches has rarely been studied. In this paper, we investigate the effectiveness of CAV testing grounds by its capability to recreate real-world traffic scenarios. We extract typical use cases from naturalistic driving events leveraging non-parametric Bayesian learning techniques. Then, we contribute to a generative sample-based optimization approach to assess the compatibility between traffic scenarios and testing ground road structure. We evaluate the effectiveness of our approach with three CAV testing facilities: Mcity, Almono (Uber ATG), and Kcity. Experiments show that our approach is effective in evaluating the capability of a given CAV testing ground to accommodate real-world driving scenarios.

[1]  Philip Koopman,et al.  Challenges in Autonomous Vehicle Testing and Validation , 2016 .

[2]  Julius Ziegler,et al.  Lanelets: Efficient map representation for autonomous driving , 2014, 2014 IEEE Intelligent Vehicles Symposium Proceedings.

[3]  Li Li,et al.  Intelligence Testing for Autonomous Vehicles: A New Approach , 2016, IEEE Transactions on Intelligent Vehicles.

[4]  István Varga,et al.  Development of a Test Track for Driverless Cars: Vehicle Design, Track Configuration, and Liability Considerations , 2017 .

[5]  Stephen J. McKenna,et al.  Tracking human motion using auxiliary particle filters and iterated likelihood weighting , 2007, Image Vis. Comput..

[6]  Heinrich Daembkes,et al.  Automated Driving Safer and More Efficient Future Driving Foreword , 2017 .

[7]  G. Kitagawa.  Monte Carlo Filter and Smoother for Non-Gaussian Nonlinear State Space Models , 1996 .

[8]  Eamonn J. Keogh,et al.  Searching and Mining Trillions of Time Series Subsequences under Dynamic Time Warping , 2012, KDD.

[9]  Hermann Winner,et al.  Functional Decomposition: An Approach to Reduce the Approval Effort for Highly Automated Driving , 2017 .

[10]  Matthias Mayr,et al.  Lanelet2: A high-definition map framework for the future of automated driving , 2018, 2018 21st International Conference on Intelligent Transportation Systems (ITSC).

[11]  Hui Ding,et al.  Querying and mining of time series data: experimental comparison of representations and distance measures , 2008, Proc. VLDB Endow..

[12]  John J. Leonard,et al.  Past, Present, and Future of Simultaneous Localization and Mapping: Toward the Robust-Perception Age , 2016, IEEE Transactions on Robotics.

[13]  Steven E Shladover,et al.  Development of California Regulations to Govern Testing and Operation of Automated Driving Systems , 2015 .

[14]  Hugh F. Durrant-Whyte,et al.  Simultaneous localization and mapping: part I , 2006, IEEE Robotics & Automation Magazine.

[15]  Yuan Zhang,et al.  Validation of a FEA Tire Model for Vehicle Dynamic Analysis and Full Vehicle Real Time Proving Ground Simulations , 1997 .

[16]  Michael I. Jordan,et al.  A Sticky HDP-HMM With Application to Speaker Diarization , 2009, 0905.2592.

[17]  Wenshuo Wang,et al.  Clustering of Driving Scenarios Using Connected Vehicle Datasets , 2018, ArXiv.

[18]  Nanning Zheng,et al.  Task-Specific Performance Evaluation of UGVs: Case Studies at the IVFC , 2014, IEEE Transactions on Intelligent Transportation Systems.

[19]  Huei Peng,et al.  Accelerated Evaluation of Automated Vehicles in Lane Change Scenarios , 2015 .

[20]  Wolfram Burgard,et al.  Monte Carlo localization for mobile robots , 1999, Proceedings 1999 IEEE International Conference on Robotics and Automation (Cat. No.99CH36288C).

[21]  Nanning Zheng,et al.  Artificial intelligence test: a case study of intelligent vehicles , 2018, Artificial Intelligence Review.

[22]  Wenshuo Wang,et al.  Extracting Traffic Primitives Directly From Naturalistically Logged Data for Self-Driving Applications , 2017, IEEE Robotics and Automation Letters.

[23]  Simon J. Godsill,et al.  On sequential simulation-based methods for Bayesian filtering , 1998 .

[24]  Scott Kirsch,et al.  Proving grounds , 2005 .

[25]  Hermann Winner,et al.  Use Cases for Autonomous Driving , 2016 .

[26]  Martijn Tideman,et al.  An Integral Approach to Autonomous and Cooperative Vehicles Development and Testing , 2015, 2015 IEEE 18th International Conference on Intelligent Transportation Systems.

[27]  Wenshuo Wang,et al.  Understanding V2V Driving Scenarios through Traffic Primitives , 2018, ArXiv.

[28]  Aled Williams,et al.  The European New Car Assessment Programme: A historical review , 2016, Chinese journal of traumatology = Zhonghua chuang shang za zhi.

[29]  Philip Chan,et al.  Toward accurate dynamic time warping in linear time and space , 2007, Intell. Data Anal..

[30]  John P. Snyder,et al.  Map Projections: A Working Manual , 2012 .

[31]  Junqiang Xi,et al.  Driving Style Analysis Using Primitive Driving Patterns With Bayesian Nonparametric Approaches , 2017, IEEE Transactions on Intelligent Transportation Systems.

[32]  Wenshuo Wang,et al.  Clustering of Driving Encounter Scenarios Using Connected Vehicle Trajectories , 2018, IEEE Transactions on Intelligent Vehicles.

[33]  N. Gordon,et al.  Novel approach to nonlinear/non-Gaussian Bayesian state estimation , 1993 .