An Application-Driven Conceptualization of Corner Cases for Perception in Highly Automated Driving

Systems and functions that rely on machine learning (ML) are the basis of highly automated driving. An essential task of such ML models is to reliably detect and interpret unusual, new, and potentially dangerous situations. The detection of those situations, which we refer to as corner cases, is highly relevant for successfully developing, applying, and validating automotive perception functions in future vehicles where multiple sensor modalities will be used. A complication for the development of corner case detectors is the lack of consistent definitions, terms, and corner case descriptions, especially when taking into account various automotive sensors. In this work, we provide an application-driven view of corner cases in highly automated driving. To achieve this goal, we first consider existing definitions of the general outlier, novelty, anomaly, and out-of-distribution detection to show relations and differences to corner cases. Moreover, we extend an existing camera-focused systematization of corner cases by adding RADAR (radio detection and ranging) and LiDAR (light detection and ranging) sensors. For this, we describe an exemplary toolchain for data acquisition and processing, highlighting the interfaces of corner case detection. We also define a novel level of corner cases, the method layer corner cases, which appear due to uncertainty inherent in the methodology.

[1]  Sebastian Ramos,et al.  Lost and Found: detecting small road hazards for self-driving vehicles , 2016, 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[2]  Tim Fingscheidt,et al.  Towards Corner Case Detection for Autonomous Driving , 2019, 2019 IEEE Intelligent Vehicles Symposium (IV).

[3]  Lennart Svensson,et al.  LIDAR-Camera Fusion for Road Detection Using Fully Convolutional Neural Networks , 2018, Robotics Auton. Syst..

[4]  Antonia Breuer,et al.  openDD: A Large-Scale Roundabout Drone Dataset , 2020, 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC).

[5]  Jishnu Mukhoti,et al.  Evaluating Bayesian Deep Learning Methods for Semantic Segmentation , 2018, ArXiv.

[6]  Paul Newman,et al.  1 year, 1000 km: The Oxford RobotCar dataset , 2017, Int. J. Robotics Res..

[7]  Tim Fingscheidt,et al.  Corner Cases for Visual Perception in Automated Driving: Some Guidance on Detection Approaches , 2021, ArXiv.

[8]  Nikolaos Doulamis,et al.  Stacked Autoencoders for Outlier Detection in Over-the-Horizon Radar Signals , 2017, Comput. Intell. Neurosci..

[9]  Qiang Xu,et al.  nuScenes: A Multimodal Dataset for Autonomous Driving , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Andreas Geiger,et al.  Vision meets robotics: The KITTI dataset , 2013, Int. J. Robotics Res..

[11]  Brian Rigling,et al.  SV-Means: A Fast SVM-Based Level Set Estimator for Phase-Modulated Radar Waveform Classification , 2018, IEEE Journal of Selected Topics in Signal Processing.

[12]  Sebastian Ramos,et al.  The Cityscapes Dataset for Semantic Urban Scene Understanding , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Roland Siegwart,et al.  The Fishyscapes Benchmark: Measuring Blind Spots in Semantic Segmentation , 2019, International Journal of Computer Vision.

[14]  P PopoolaOluwatoyin,et al.  Video-Based Abnormal Human Behavior Recognition—A Review , 2012 .

[15]  Leonidas J. Guibas,et al.  Frustum PointNets for 3D Object Detection from RGB-D Data , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[16]  Sergio Casas,et al.  RadarNet: Exploiting Radar for Robust Perception of Dynamic Objects , 2020, ECCV.

[17]  Raquel Urtasun,et al.  Identifying Unknown Instances for Autonomous Driving , 2019, CoRL.

[18]  Philip Koopman,et al.  Credible Autonomy Safety Argumentation , 2018 .

[19]  Véronique Berge-Cherfaoui,et al.  Evidential deep learning for arbitrary LIDAR object classification in the context of autonomous driving , 2019, 2019 IEEE Intelligent Vehicles Symposium (IV).

[20]  Michael Meyer,et al.  Automotive Radar Dataset for Deep Learning Based 3D Object Detection , 2019, 2019 16th European Radar Conference (EuRAD).

[21]  Tim Fingscheidt,et al.  Systematization of Corner Cases for Visual Perception in Automated Driving , 2020, 2020 IEEE Intelligent Vehicles Symposium (IV).

[22]  Andrew M. Wallace,et al.  RADIATE: A Radar Dataset for Automotive Perception in Bad Weather , 2020, 2021 IEEE International Conference on Robotics and Automation (ICRA).

[23]  Joan Bruna,et al.  Intriguing properties of neural networks , 2013, ICLR.

[24]  Jonathon Shlens,et al.  Explaining and Harnessing Adversarial Examples , 2014, ICLR.

[25]  Gonzalo Farias,et al.  A Neural Network Approach for Building An Obstacle Detection Model by Fusion of Proximity Sensors Data , 2018, Sensors.

[26]  Yohannes Kassahun,et al.  A2D2: Audi Autonomous Driving Dataset , 2020, ArXiv.

[27]  B. Sick,et al.  Towards Corner Case Detection by Modeling the Uncertainty of Instance Segmentation Networks , 2020, ICPR Workshops.

[28]  Bernhard Sick,et al.  Novelty detection in continuously changing environments , 2021, Future Gener. Comput. Syst..

[29]  Hyun Chul Roh,et al.  Complex urban dataset with multi-level sensors from highly diverse urban environments , 2019, Int. J. Robotics Res..

[30]  Zoubin Ghahramani,et al.  Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning , 2015, ICML.

[31]  Pascal Fua,et al.  Detecting the Unexpected via Image Resynthesis , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[32]  Aggelos K. Katsaggelos,et al.  Video anomaly detection in spatiotemporal context , 2010, 2010 IEEE International Conference on Image Processing.

[33]  Peter Kontschieder,et al.  The Mapillary Vistas Dataset for Semantic Understanding of Street Scenes , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[34]  Kejun Wang,et al.  Video-Based Abnormal Human Behavior Recognition—A Review , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[35]  Klaus C. J. Dietmayer,et al.  2D Car Detection in Radar Data with PointNets , 2019, 2019 IEEE Intelligent Transportation Systems Conference (ITSC).

[36]  M. Rottmann,et al.  Entropy Maximization and Meta Classification for Out-of-Distribution Detection in Semantic Segmentation , 2020, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[37]  Klaus C. J. Dietmayer,et al.  Deep Multi-Modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges , 2019, IEEE Transactions on Intelligent Transportation Systems.

[38]  Mohamed-Cherif Rahal,et al.  Real Time Lidar and Radar High-Level Fusion for Obstacle Detection and Tracking with evaluation on a ground truth , 2018, ArXiv.

[39]  Anne M. Pavy,et al.  Open-Set Radar Waveform Classification: Comparison of Different Features and Classifiers , 2020, 2020 IEEE International Radar Conference (RADAR).

[40]  Charles Blundell,et al.  Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles , 2016, NIPS.

[41]  Krzysztof Czarnecki,et al.  Canadian Adverse Driving Conditions dataset , 2020, Int. J. Robotics Res..

[42]  Alex Kendall,et al.  What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision? , 2017, NIPS.

[43]  VARUN CHANDOLA,et al.  Anomaly detection: A survey , 2009, CSUR.

[44]  Ruigang Yang,et al.  The ApolloScape Dataset for Autonomous Driving , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[45]  Dariu Gavrila,et al.  EuroCity Persons: A Novel Benchmark for Person Detection in Traffic Scenes , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[46]  David A. Clifton,et al.  A review of novelty detection , 2014, Signal Process..