An Active Learning Reliability Method for Systems with Partially Defined Performance Functions

In engineering design, one often wishes to calculate the probability that the performance of a system is satisfactory under uncertainty. State of the art algorithms exist to solve this problem using active learning with Gaussian process models. However, these algorithms cannot be applied to problems which often occur in the autonomous vehicle domain where the performance of a system may be undefined under certain circumstances. To solve this problem, we introduce a hierarchical model for the system performance, where undefined performance is classified before the performance is regressed. This enables active learning Gaussian process methods to be applied to problems where the performance of the system is some-times undefined, and we demonstrate the effectiveness of our approach by testing our methodology on synthetic numerical examples for the autonomous driving domain. The code to generate these experiments is available as an open source repository: https://github.com/fiveai/hGP_experiments/

[1]  Matthias G. R. Faes,et al.  Parallel adaptive Bayesian quadrature for rare event estimation , 2022, Reliab. Eng. Syst. Saf..

[2]  Mark Pullin,et al.  Emulation of physical processes with Emukit , 2021, ArXiv.

[3]  A. Eichberger,et al.  Scenario Based Testing of Automated Driving Systems: A Literature Survey , 2021, FISITA World Congress 2021 - Technical Programme.

[4]  Alan O'Connor,et al.  Adaptive approaches in metamodel-based reliability analysis: A review , 2021, Structural Safety.

[5]  Ichiro Takeuchi,et al.  Active learning for distributionally robust level-set estimation , 2021, ICML.

[6]  R. Urtasun,et al.  AdvSim: Generating Safety-Critical Scenarios for Self-Driving Vehicles , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Ichiro Takeuchi,et al.  Bayesian Quadrature Optimization for Probability Threshold Robustness Measure , 2020, Neural Computation.

[8]  S. Marelli,et al.  A generalized framework for active learning reliability: survey and benchmark , 2021 .

[9]  Bruno Sudret,et al.  A TWO-STAGE SURROGATE MODELING APPROACH FOR THE APPROXIMATION OF MODELS WITH NON-SMOOTH OUTPUTS , 2019, Proceedings of the 3rd International Conference on Uncertainty Quantification in Computational Sciences and Engineering (UNCECOMP 2019).

[10]  Pushmeet Kohli,et al.  Rigorous Agent Evaluation: An Adversarial Approach to Uncover Catastrophic Failures , 2018, ICLR.

[11]  Zhenzhou Lu,et al.  AK-SYSi: an improved adaptive Kriging model for system reliability analysis with multiple failure modes by a refined U learning function , 2018, Structural and Multidisciplinary Optimization.

[12]  Mykel J. Kochenderfer,et al.  Adaptive Stress Testing for Autonomous Vehicles , 2018, 2018 IEEE Intelligent Vehicles Symposium (IV).

[13]  Philip Koopman,et al.  Toward a Framework for Highly Automated Vehicle Safety Validation , 2018 .

[14]  S. M. Sohel Mahmud,et al.  Application of proximal surrogate indicators for safety evaluation: A review of recent developments and research needs , 2017 .

[15]  Michael Stolz,et al.  Testing of autonomous vehicles using surrogate models and stochastic optimization , 2017, 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC).

[16]  Philip Koopman,et al.  Challenges in Autonomous Vehicle Testing and Validation , 2016 .

[17]  Nicolas Gayton,et al.  AK-MCS: An active learning reliability method combining Kriging and Monte Carlo Simulation , 2011 .

[18]  Trevor Darrell,et al.  Active Learning with Gaussian Processes for Object Categorization , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[19]  Bernhard Sendhoff,et al.  Robust Optimization - A Comprehensive Survey , 2007 .

[20]  Christopher K. I. Williams,et al.  Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning) , 2005 .

[21]  Robert E. Melchers,et al.  Structural Reliability: Analysis and Prediction , 1987 .