Validation Frameworks for Self-Driving Vehicles: A Survey

As a part of the digital transformation, we interact with more and more intelligent gadgets. Today, these gadgets are often mobile devices, but in the advent of smart cities, more and more infrastructure---such as traffic and buildings---in our surroundings becomes intelligent. The intelligence, however, does not emerge by itself. Instead, we need both design techniques to create intelligent systems, as well as approaches to validate their correct behavior. An example of intelligent systems that could benefit smart cities are self-driving vehicles. Self-driving vehicles are continuously becoming both commercially available and common on roads. Accidents involving self-driving vehicles, however, have raised concerns about their reliability. Due to these concerns, the safety of self-driving vehicles should be thoroughly tested before they can be released into traffic. To ensure that self-driving vehicles encounter all possible scenarios, several millions of hours of testing must be carried out; therefore, testing self-driving vehicles in the real world is impractical. There is also the issue that testing self-driving vehicles directly in the traffic poses a potential safety hazard to human drivers. To tackle this challenge, validation frameworks for testing self-driving vehicles in simulated scenarios are being developed by academia and industry. In this chapter, we briefly introduce self-driving vehicles and give an overview of validation frameworks for testing them in a simulated environment. We conclude by discussing what an ideal validation framework at the state of the art should be and what could benefit validation frameworks for self-driving vehicles in the future.

[1]  Juan-Carlos Cano,et al.  V2X-d: A vehicular density estimation system that combines V2V and V2I communications , 2013, 2013 IFIP Wireless Days (WD).

[2]  Andrew Howard,et al.  Design and use paradigms for Gazebo, an open-source multi-robot simulator , 2004, 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Cat. No.04CH37566).

[3]  Philip Koopman,et al.  Challenges in Autonomous Vehicle Testing and Validation , 2016 .

[4]  Mike Lukuc,et al.  Vehicle-to-Vehicle Communications: Readiness of V2V Technology for Application , 2014 .

[5]  Ravishankar K. Iyer,et al.  AVFI: Fault Injection for Autonomous Vehicles , 2018, 2018 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks Workshops (DSN-W).

[6]  Sarfraz Khurshid,et al.  DeepRoad: GAN-Based Metamorphic Testing and Input Validation Framework for Autonomous Driving Systems , 2018, 2018 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE).

[7]  Rob Fergus,et al.  Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks , 2015, NIPS.

[8]  Bowen Zheng,et al.  Next Generation Automotive Architecture Modeling and Exploration for Autonomous Driving , 2016, 2016 IEEE Computer Society Annual Symposium on VLSI (ISVLSI).

[9]  Ravishankar K. Iyer,et al.  Kayotee: A Fault Injection-based System to Assess the Safety and Reliability of Autonomous Vehicles to Faults and Errors , 2019, ArXiv.

[10]  Nan Li,et al.  Game Theoretic Modeling of Driver and Vehicle Interactions for Verification and Validation of Autonomous Vehicle Control Systems , 2016, IEEE Transactions on Control Systems Technology.

[11]  Saeid Nahavandi,et al.  A Classifier Graph Based Recurring Concept Detection and Prediction Approach , 2018, Comput. Intell. Neurosci..

[12]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[13]  Joshué Pérez,et al.  A complete framework for developing and testing automated driving controllers , 2017 .

[14]  Mark H. Overmars,et al.  A Comparative Study of Probabilistic Roadmap Planners , 2002, WAFR.

[15]  Steven E. Shladover,et al.  Potential Cyberattacks on Automated Vehicles , 2015, IEEE Transactions on Intelligent Transportation Systems.

[16]  Dong Ngoduy,et al.  Enhanced cooperative car-following traffic model with the combination of V2V and V2I communication , 2016 .

[17]  I. Glickstein,et al.  Route Planning , 2017, Encyclopedia of GIS.

[18]  Bowen Zheng,et al.  CONVINCE: A cross-layer modeling, exploration and validation framework for next-generation connected vehicles , 2016, 2016 IEEE/ACM International Conference on Computer-Aided Design (ICCAD).

[19]  Daniel Krajzewicz,et al.  Recent Development and Applications of SUMO - Simulation of Urban MObility , 2012 .

[20]  Ralf Kohlhaas,et al.  Simulation framework for the development of autonomous small scale vehicles , 2016, 2016 IEEE International Conference on Simulation, Modeling, and Programming for Autonomous Robots (SIMPAR).

[21]  Suman Jana,et al.  DeepTest: Automated Testing of Deep-Neural-Network-Driven Autonomous Cars , 2017, 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE).

[22]  Ajmal Mian,et al.  Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Survey , 2018, IEEE Access.

[23]  Philip Koopman,et al.  A Safety Standard Approach for Fully Autonomous Vehicles , 2019, SAFECOMP Workshops.

[24]  Myoungho Sunwoo,et al.  Development of Autonomous Car—Part II: A Case Study on the Implementation of an Autonomous Driving System Based on Distributed Architecture , 2015, IEEE Transactions on Industrial Electronics.

[25]  Dipak Ghosal,et al.  Security vulnerabilities of connected vehicle streams and their impact on cooperative driving , 2015, IEEE Communications Magazine.

[26]  Miguel A. Costa-Gomes,et al.  Comparing Models of Strategic Thinking in Van Huyck, Battalio, and Beil’s Coordination Games , 2009 .

[27]  Homa Alemzadeh,et al.  Experimental Resilience Assessment of an Open-Source Driving Agent , 2018, 2018 IEEE 23rd Pacific Rim International Symposium on Dependable Computing (PRDC).

[28]  Tommi Mikkonen,et al.  AILiveSim: An Extensible Virtual Environment for Training Autonomous Vehicles , 2019, 2019 IEEE 43rd Annual Computer Software and Applications Conference (COMPSAC).

[29]  Thomas Engel,et al.  A Car Hacking Experiment: When Connectivity Meets Vulnerability , 2015, 2015 IEEE Globecom Workshops (GC Wkshps).

[30]  Jin-Hee Cho,et al.  Automotive hardware development according to ISO 26262 , 2011, 13th International Conference on Advanced Communication Technology (ICACT2011).

[31]  Nicolas Hiblot,et al.  Pro-SiVIC and ROADS, a software suite for sensors simulation and virtual prototyping of ADAS , 2010 .

[32]  Dimitris N. Metaxas,et al.  StackGAN: Text to Photo-Realistic Image Synthesis with Stacked Generative Adversarial Networks , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[33]  Ravishankar K. Iyer,et al.  ML-Based Fault Injection for Autonomous Vehicles: A Case for Bayesian Fault Injection , 2019, 2019 49th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN).

[34]  Junfeng Yang,et al.  DeepXplore: Automated Whitebox Testing of Deep Learning Systems , 2017, SOSP.

[35]  Urbano Nunes,et al.  Intelligent traffic management at intersections supported by V2V and V2I communications , 2012, 2012 15th International IEEE Conference on Intelligent Transportation Systems.

[36]  Taxonomy and definitions for terms related to driving automation systems for on-road motor vehicles , 2022 .

[37]  Vrizlynn L. L. Thing,et al.  Autonomous vehicle ultrasonic sensor vulnerability and impact assessment , 2018, 2018 IEEE 4th World Forum on Internet of Things (WF-IoT).

[38]  I. Kougias,et al.  How Can Autonomous and Connected Vehicles, Electromobility, BRT, Hyperloop, Shared Use Mobility and Mobility-As-A-Service Shape Transport Futures for the Context of Smart Cities? , 2017 .

[39]  Miguel A. Costa-Gomes,et al.  Cognition and Behavior in Two-Person Guessing Games: An Experimental Study , 2003 .

[40]  John O. Clark System of Systems Engineering and Family of Systems Engineering from a standards, V-Model, and Dual-V Model perspective , 2009, 2009 3rd Annual IEEE Systems Conference.

[41]  Gerhard Grießnig,et al.  Development of the ISO 21448 , 2019, EuroSPI.

[42]  T. Hedden,et al.  What do you think I think you think?: Strategic reasoning in matrix games , 2002, Cognition.

[43]  D. Stahl,et al.  On Players' Models of Other Players: Theory and Experimental Evidence , 1995 .

[44]  Thomas Schamm,et al.  Testing and validating high level components for automated driving: simulation framework for traffic scenarios , 2016, 2016 IEEE Intelligent Vehicles Symposium (IV).

[45]  Nikolaos Doulamis,et al.  Deep Learning for Computer Vision: A Brief Review , 2018, Comput. Intell. Neurosci..