Uncertainty Evaluation of Object Detection Algorithms for Autonomous Vehicles

The safety of the intended functionality (SOTIF) has become one of the hottest topics in the field of autonomous driving. However, no testing and evaluating system for SOTIF performance has been proposed yet. Therefore, this paper proposes a framework based on the advanced You Only Look Once (YOLO) algorithm and the mean Average Precision (mAP) method to evaluate the object detection performance of the camera under SOTIF-related scenarios. First, a dataset is established, which contains road images with extreme weather and adverse lighting conditions. Second, the Monte Carlo dropout (MCD) method is used to analyze the uncertainty of the algorithm and draw the uncertainty region of the predicted bounding box. Then, the confidence of the algorithm is calibrated based on uncertainty results so that the average confidence after calibration can better reflect the real accuracy. The uncertainty results and the calibrated confidence are proposed to be used for online risk identification. Finally, the confusion matrix is extended according to the several possible mistakes that the object detection algorithm may make, and then the mAP is calculated as an index for offline evaluation and comparison. This paper offers suggestions to apply the MCD method to complex object detection algorithms and to find the relationship between the uncertainty and the confidence of the algorithm. The experimental results verified by specific SOTIF scenarios proof the feasibility and effectiveness of the proposed uncertainty acquisition approach for object detection algorithm, which provides potential practical implementation chance to address perceptual related SOTIF risk for autonomous vehicles.

[1]  Roberto Cipolla,et al.  Bayesian SegNet: Model Uncertainty in Deep Convolutional Encoder-Decoder Architectures for Scene Understanding , 2015, BMVC.

[2]  Jack Weast,et al.  Literature Review of Verification and Validation Activities of Automated Driving Systems , 2021 .

[3]  Nitish Srivastava,et al.  Improving neural networks by preventing co-adaptation of feature detectors , 2012, ArXiv.

[4]  Luca Cardelli,et al.  Uncertainty Quantification with Statistical Guarantees in End-to-End Autonomous Driving Control , 2019, 2020 IEEE International Conference on Robotics and Automation (ICRA).

[5]  Yanjun Huang,et al.  Ethical Decision-Making Platform in Autonomous Vehicles With Lexicographic Optimization Based Model Predictive Controller , 2020, IEEE Transactions on Vehicular Technology.

[6]  Niko Sünderhauf,et al.  Benchmarking Sampling-based Probabilistic Object Detectors , 2019, CVPR Workshops.

[7]  Pietro Perona,et al.  Microsoft COCO: Common Objects in Context , 2014, ECCV.

[8]  Michael Milford,et al.  Evaluating Merging Strategies for Sampling-based Uncertainty Techniques in Object Detection , 2018, 2019 International Conference on Robotics and Automation (ICRA).

[9]  Yu Sun,et al.  Evaluation of machine learning methods to stroke outcome prediction using a nationwide disease registry , 2020, Comput. Methods Programs Biomed..

[10]  Sergey Levine,et al.  Uncertainty-Aware Reinforcement Learning for Collision Avoidance , 2017, ArXiv.

[11]  Zoubin Ghahramani,et al.  Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning , 2015, ICML.

[12]  Tiago Azevedo,et al.  Stochastic-YOLO: Efficient Probabilistic Object Detection under Dataset Shifts , 2020, ArXiv.

[13]  Xiao-Li Meng,et al.  Perfection within Reach: Exact MCMC Sampling , 2011 .

[14]  Bianca Zadrozny,et al.  Obtaining calibrated probability estimates from decision trees and naive Bayesian classifiers , 2001, ICML.

[15]  Ling Shao,et al.  Performance evaluation of deep feature learning for RGB-D image/video classification , 2017, Inf. Sci..

[16]  Andreas Geiger,et al.  Vision meets robotics: The KITTI dataset , 2013, Int. J. Robotics Res..

[17]  Zoubin Ghahramani,et al.  A Theoretically Grounded Application of Dropout in Recurrent Neural Networks , 2015, NIPS.

[18]  Milos Hauskrecht,et al.  Obtaining Well Calibrated Probabilities Using Bayesian Binning , 2015, AAAI.

[19]  Nikolay Laptev,et al.  Deep and Confident Prediction for Time Series at Uber , 2017, 2017 IEEE International Conference on Data Mining Workshops (ICDMW).

[20]  Amir Khajepour,et al.  Ethical Decision Making in Autonomous Vehicles: Challenges and Research Progress , 2022, IEEE Intelligent Transportation Systems Magazine.

[21]  Hong Wang,et al.  Crash Mitigation in Motion Planning for Autonomous Vehicles , 2019, IEEE Transactions on Intelligent Transportation Systems.

[22]  Julien Cornebise,et al.  Weight Uncertainty in Neural Networks , 2015, ArXiv.

[23]  Kilian Q. Weinberger,et al.  On Calibration of Modern Neural Networks , 2017, ICML.

[24]  Xiaojie Guo,et al.  Fast Single Image Rain Removal via a Deep Decomposition-Composition Network , 2018, CVPR 2018.

[25]  Niko Sünderhauf,et al.  Dropout Sampling for Robust Object Detection in Open-Set Conditions , 2017, 2018 IEEE International Conference on Robotics and Automation (ICRA).