How to Evaluate Proving Grounds for Self-Driving? A Quantitative Approach

Proving ground has been a critical component in testing and validation for Connected and Automated Vehicles (CAV). Although quite a few world-class testing facilities have been under construction over the years, the evaluation of proving grounds themselves as testing approaches has rarely been studied. In this paper, we present the first attempt to systematically evaluate CAV proving grounds and contribute to a generative sample-based approach to assessing the representation of traffic scenarios in proving grounds. Leveraging typical use cases extracted from naturalistic driving events, we establish a strong link between proving ground testing results of CAVs and their anticipated public street performance. We present benchmark results of our approach on three world-class CAV testing facilities: Mcity, Almono (Uber ATG), and Kcity. We successfully show the overall evaluation of these proving grounds in terms of their capability to accommodate real-world traffic scenarios. We believe that when the effectiveness of a testing ground itself is validated, the testing results would grant more confidence for CAV public deployment.

[1]  Huei Peng,et al.  Accelerated Evaluation of Automated Vehicles in Lane Change Scenarios , 2015 .

[2]  Steven E Shladover,et al.  Development of California Regulations to Govern Testing and Operation of Automated Driving Systems , 2015 .

[3]  N. Gordon,et al.  Novel approach to nonlinear/non-Gaussian Bayesian state estimation , 1993 .

[4]  Nanning Zheng,et al.  Artificial intelligence test: a case study of intelligent vehicles , 2018, Artificial Intelligence Review.

[5]  Simon J. Godsill,et al.  On sequential simulation-based methods for Bayesian filtering , 1998 .

[6]  Matthias Mayr,et al.  Lanelet2: A high-definition map framework for the future of automated driving , 2018, 2018 21st International Conference on Intelligent Transportation Systems (ITSC).

[7]  P. Liu,et al.  Road tests of self-driving vehicles: Affective and cognitive pathways in acceptance formation , 2019, Transportation Research Part A: Policy and Practice.

[8]  Stephen J. McKenna,et al.  Tracking human motion using auxiliary particle filters and iterated likelihood weighting , 2007, Image Vis. Comput..

[9]  Eamonn J. Keogh,et al.  Searching and Mining Trillions of Time Series Subsequences under Dynamic Time Warping , 2012, KDD.

[10]  G. Kitagawa.  Monte Carlo Filter and Smoother for Non-Gaussian Nonlinear State Space Models , 1996 .

[11]  Wolfram Burgard,et al.  Monte Carlo localization for mobile robots , 1999, Proceedings 1999 IEEE International Conference on Robotics and Automation (Cat. No.99CH36288C).

[12]  Scott Kirsch,et al.  Proving grounds , 2005 .

[13]  Philip Koopman,et al.  Challenges in Autonomous Vehicle Testing and Validation , 2016 .

[14]  Julius Ziegler,et al.  Lanelets: Efficient map representation for autonomous driving , 2014, 2014 IEEE Intelligent Vehicles Symposium Proceedings.

[15]  Björn Schünemann,et al.  V2X simulation runtime infrastructure VSimRTI: An assessment tool to design smart traffic management systems , 2011, Comput. Networks.

[16]  Li Li,et al.  Intelligence Testing for Autonomous Vehicles: A New Approach , 2016, IEEE Transactions on Intelligent Vehicles.

[17]  Wenshuo Wang,et al.  Clustering of Driving Encounter Scenarios Using Connected Vehicle Trajectories , 2018, IEEE Transactions on Intelligent Vehicles.

[18]  John P. Snyder,et al.  Map Projections: A Working Manual , 2012 .

[19]  Wenshuo Wang,et al.  Understanding V2V Driving Scenarios through Traffic Primitives , 2018, ArXiv.

[20]  Hugh F. Durrant-Whyte,et al.  Simultaneous localization and mapping: part I , 2006, IEEE Robotics & Automation Magazine.

[21]  Heinrich Daembkes,et al.  Automated Driving Safer and More Efficient Future Driving Foreword , 2017 .

[22]  Hermann Winner,et al.  Use Cases for Autonomous Driving , 2016 .

[23]  Aled Williams,et al.  The European New Car Assessment Programme: A historical review , 2016, Chinese journal of traumatology = Zhonghua chuang shang za zhi.

[24]  John J. Leonard,et al.  Past, Present, and Future of Simultaneous Localization and Mapping: Toward the Robust-Perception Age , 2016, IEEE Transactions on Robotics.

[25]  Philip Chan,et al.  Toward accurate dynamic time warping in linear time and space , 2007, Intell. Data Anal..

[26]  Hui Ding,et al.  Querying and mining of time series data: experimental comparison of representations and distance measures , 2008, Proc. VLDB Endow..

[27]  Nanning Zheng,et al.  Task-Specific Performance Evaluation of UGVs: Case Studies at the IVFC , 2014, IEEE Transactions on Intelligent Transportation Systems.

[28]  István Varga,et al.  Development of a Test Track for Driverless Cars: Vehicle Design, Track Configuration, and Liability Considerations , 2017 .

[29]  Martijn Tideman,et al.  An Integral Approach to Autonomous and Cooperative Vehicles Development and Testing , 2015, 2015 IEEE 18th International Conference on Intelligent Transportation Systems.

[30]  Junqiang Xi,et al.  Driving Style Analysis Using Primitive Driving Patterns With Bayesian Nonparametric Approaches , 2017, IEEE Transactions on Intelligent Transportation Systems.

[31]  Michael I. Jordan,et al.  A Sticky HDP-HMM With Application to Speaker Diarization , 2009, 0905.2592.

[32]  Yuan Zhang,et al.  Validation of a FEA Tire Model for Vehicle Dynamic Analysis and Full Vehicle Real Time Proving Ground Simulations , 1997 .

[33]  Hermann Winner,et al.  Functional Decomposition: An Approach to Reduce the Approval Effort for Highly Automated Driving , 2017 .

[34]  Wenshuo Wang,et al.  Extracting Traffic Primitives Directly From Naturalistically Logged Data for Self-Driving Applications , 2017, IEEE Robotics and Automation Letters.