Handling Noise in Search-Based Scenario Generation for Autonomous Driving Systems

This paper presents the first evaluation of k-nearest neighbours-Averaging (kNN-Avg) on a real-world case study. kNN-Avg is a novel technique that tackles the challenges of noisy multi-objective optimisation (MOO). Existing studies suggest the use of repetition to overcome noise. In contrast, kNN-Avg approximates these repetitions and exploits previous executions, thereby avoiding the cost of re-running. We use kNN-Avg for the scenario generation of a real-world autonomous driving system (ADS) and show that it is better than the noisy baseline. Furthermore, we compare it to the repetition-method and outline indicators as to which approach to choose in which situations.

[1]  Lothar Thiele,et al.  Comparison of Multiobjective Evolutionary Algorithms: Empirical Results , 2000, Evolutionary Computation.

[2]  Kay Chen Tan,et al.  Evolutionary Multi-objective Optimization in Uncertain Environments - Issues and Algorithms , 2009, Studies in Computational Intelligence.

[3]  Claire Le Goues,et al.  A Study on Challenges of Testing Robotic Systems , 2020, 2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST).

[4]  Lionel C. Briand,et al.  Testing advanced driver assistance systems using multi-objective search and neural networks , 2016, 2016 31st IEEE/ACM International Conference on Automated Software Engineering (ASE).

[5]  Kalyanmoy Deb,et al.  A fast and elitist multiobjective genetic algorithm: NSGA-II , 2002, IEEE Trans. Evol. Comput..

[6]  Kay Chen Tan,et al.  An Investigation on Noisy Environments in Evolutionary Multiobjective Optimization , 2007, IEEE Transactions on Evolutionary Computation.

[7]  Magnus C. Ohlsson,et al.  Experimentation in Software Engineering , 2000, The Kluwer International Series in Software Engineering.

[8]  Jürgen Teich,et al.  Pareto-Front Exploration with Uncertain Objectives , 2001, EMO.

[9]  Lionel C. Briand,et al.  A practical guide for using statistical tests to assess randomized algorithms in software engineering , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[10]  Paolo Arcaini,et al.  KNN-Averaging for Noisy Multi-objective Optimisation , 2021, QUATIC.

[11]  J. Fitzpatrick,et al.  Genetic Algorithms in Noisy Environments , 2005, Machine Learning.

[12]  Kalyanmoy Deb,et al.  Pymoo: Multi-Objective Optimization in Python , 2020, IEEE Access.

[13]  Xin Yao,et al.  of Birmingham Quality evaluation of solution sets in multiobjective optimisation , 2019 .

[14]  Paolo Arcaini,et al.  Generating Avoidable Collision Scenarios for Testing Autonomous Driving Systems , 2020, 2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST).

[15]  Saurabh Jha,et al.  AV-FUZZER: Finding Safety Violations in Autonomous Driving Systems , 2020, 2020 IEEE 31st International Symposium on Software Reliability Engineering (ISSRE).

[16]  Jonathan E. Fieldsend,et al.  Multi-objective optimisation in the presence of uncertainty , 2005, 2005 IEEE Congress on Evolutionary Computation.

[17]  A. E. Eiben,et al.  Introduction to Evolutionary Computing , 2003, Natural Computing Series.

[18]  Gordon Fraser,et al.  Automatically testing self-driving cars with search-based procedural content generation , 2019, ISSTA.

[19]  Kwang Ryel Ryu,et al.  Accumulative sampling for noisy evolutionary multi-objective optimization , 2011, GECCO '11.