DNN-Based Map Deviation Detection in LiDAR Point Clouds

In this work we present a novel deep learning-based approach to detect and specify map deviations in erroneous or outdated high-definition (HD) maps using both sensor and map data as input to a deep neural network (DNN). We first present our proposed reference method for map deviation detection (MDD) utilizing a sensor-only DNN detecting traffic signs, traffic lights, and pole-like objects in LiDAR data, with deviations obtained by subsequently comparing detected objects and examined map. Second, we facilitate the object detection task by using the examined map as additional input to the network. Third, we employ a specialized MDD network to directly infer the correctness of the map input. Finally, we demonstrate the robustness of our approach for challenging scenes featuring occlusions and a reduced point density, e.g., due to heavy rain. Our code is available at https://github.com/Volkswagen/3dhd_devkit.

[1]  James Hays,et al.  Trust, but Verify: Cross-Modality Fusion for HD Map Change Detection , 2022, NeurIPS Datasets and Benchmarks.

[2]  T. Fingscheidt,et al.  3DHD CityScenes: High-Definition Maps in High-Density Point Clouds , 2022, 2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC).

[3]  B. Riveiro,et al.  Point cloud semantic segmentation of complex railway environments using deep learning , 2022, Automation in Construction.

[4]  Bisheng Yang,et al.  Street-view imagery guided street furniture inventory from mobile laser scanning point clouds , 2022, ISPRS Journal of Photogrammetry and Remote Sensing.

[5]  Xianke Lin,et al.  High-Definition Map Generation Technologies For Autonomous Driving , 2022, ArXiv.

[6]  Tim Fingscheidt,et al.  DNN-Based Recognition of Pole-Like Objects in LiDAR Point Clouds , 2021, 2021 IEEE International Intelligent Transportation Systems Conference (ITSC).

[7]  Yasser H. Khalil,et al.  LiCaNet: Further Enhancement of Joint Perception and Motion Prediction Based on Multi-Modal Fusion , 2021, IEEE Open Journal of Intelligent Transportation Systems.

[8]  A. Cenedese,et al.  Use of probabilistic graphical methods for online map validation , 2021, 2021 IEEE Intelligent Vehicles Symposium Workshops (IV Workshops).

[9]  Carsten Hasberg,et al.  Boosted Classifiers on 1D Signals and Mutual Evaluation of Independently Aligned Spatio-Semantic Feature Groups for HD Map Change Detection , 2021, 2021 IEEE Intelligent Vehicles Symposium (IV).

[10]  Jianye Liu,et al.  An improved traffic lights recognition algorithm for autonomous driving in complex scenarios , 2021, Int. J. Distributed Sens. Networks.

[11]  Andreas Geiger,et al.  Multi-Modal Fusion Transformer for End-to-End Autonomous Driving , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Jingnan Liu,et al.  Real-Time HD Map Change Detection for Crowdsourcing Update Based on Mid-to-High-End Sensors , 2021, Sensors.

[13]  Chen Feng,et al.  3D Point Cloud Processing and Learning for Autonomous Driving: Impacting Map Creation, Localization, and Perception , 2021, IEEE Signal Processing Magazine.

[14]  Tim Fingscheidt,et al.  Terminology and Analysis of Map Deviations in Urban Domains: Towards Dependability for HD Maps in Automated Vehicles , 2020, 2020 IEEE Intelligent Vehicles Symposium (IV).

[15]  Gunther Krehl,et al.  Sensor Fusion-based Online Map Validation for Autonomous Driving , 2020, 2020 IEEE Intelligent Vehicles Symposium (IV).

[16]  Angelo Cenedese,et al.  Correlation-based approach to online map validation , 2020, 2020 IEEE Intelligent Vehicles Symposium (IV).

[17]  Martin Lauer,et al.  HD Map Verification Without Accurate Localization Prior Using Spatio-Semantic 1D Signals , 2020, 2020 IEEE Intelligent Vehicles Symposium (IV).

[18]  Yanan Sun,et al.  3DSSD: Point-Based 3D Single Stage Object Detector , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Charles Toth,et al.  High Definition 3D Map Creation Using GNSS/IMU/LiDAR Sensor Integration to Support Autonomous Vehicle Navigation , 2020, Sensors.

[20]  Mohsen Guizani,et al.  Deep CNN-Based Real-Time Traffic Light Detector for Self-Driving Vehicles , 2020, IEEE Transactions on Mobile Computing.

[21]  Renzhong Guo,et al.  Pole-Like Street Furniture Segmentation and Classification in Mobile LiDAR Data by Integrating Multiple Shape-Descriptor Constraints , 2019, Remote. Sens..

[22]  Weiwei Zhang,et al.  Road Marking Segmentation Based on Siamese Attention Module and Maximum Stable External Region , 2019, IEEE Access.

[23]  Wolfram Burgard,et al.  Long-Term Urban Vehicle Localization Using Pole Landmarks Extracted from 3-D Lidar Scans , 2019, 2019 European Conference on Mobile Robots (ECMR).

[24]  Ghassan Al-Regib,et al.  Traffic Sign Detection Under Challenging Conditions: A Deeper Look into Performance Variations and Spectral Characteristics , 2019, IEEE Transactions on Intelligent Transportation Systems.

[25]  Yuwei Chen,et al.  Semantic segmentation of road furniture in mobile laser scanning data , 2019, ISPRS Journal of Photogrammetry and Remote Sensing.

[26]  Xiaoyong Shen,et al.  STD: Sparse-to-Dense 3D Object Detector for Point Cloud , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[27]  Philippe Bonnifait,et al.  Estimating the reliability of georeferenced lane markings for map-aided localization , 2019, 2019 IEEE Intelligent Vehicles Symposium (IV).

[28]  Vinicius B. Cardoso,et al.  Traffic Light Recognition Using Deep Learning and Prior Maps for Autonomous Cars , 2019, 2019 International Joint Conference on Neural Networks (IJCNN).

[29]  Nanning Zheng,et al.  High-Definition Map Combined Local Motion Planning and Obstacle Avoidance for Autonomous Driving , 2019, 2019 IEEE Intelligent Vehicles Symposium (IV).

[30]  Fawzi Nashashibi,et al.  LIDAR-Based road signs detection For Vehicle Localization in an HD Map , 2019, 2019 IEEE Intelligent Vehicles Symposium (IV).

[31]  Wolfram Burgard,et al.  HD Map Change Detection with a Boosted Particle Filter , 2019, 2019 International Conference on Robotics and Automation (ICRA).

[32]  Cyrill Stachniss,et al.  Localization with Sliding Window Factor Graphs on Third-Party Maps for Automated Driving , 2019, 2019 International Conference on Robotics and Automation (ICRA).

[33]  Long Chen,et al.  Robust Lane Detection From Continuous Driving Scenes Using Deep Neural Networks , 2019, IEEE Transactions on Vehicular Technology.

[34]  Shinpei Kato,et al.  Traffic light recognition using high-definition map features , 2019, Robotics Auton. Syst..

[35]  Jiong Yang,et al.  PointPillars: Fast Encoders for Object Detection From Point Clouds , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[36]  Yi Lin,et al.  Automatic Recognition of Pole-Like Objects from Mobile Laser Scanning Point Clouds , 2018, Remote. Sens..

[37]  Bo Li,et al.  SECOND: Sparsely Embedded Convolutional Detection , 2018, Sensors.

[38]  Zhenwei Shi,et al.  Voxel-Based Extraction and Classification of 3-D Pole-Like Objects From Mobile LiDAR Point Cloud Data , 2018, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[39]  Klaus C. J. Dietmayer,et al.  Detecting Traffic Lights by Single Shot Detection , 2018, 2018 21st International Conference on Intelligent Transportation Systems (ITSC).

[40]  Zhidong Deng,et al.  Detection and Recognition of Traffic Planar Objects Using Colorized Laser Scan and Perspective Distortion Rectification , 2018, IEEE Transactions on Intelligent Transportation Systems.

[41]  Liang Zhong,et al.  Robust Traffic-Sign Detection and Classification Using Mobile LiDAR Data With Digital Images , 2018, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[42]  Belén Riveiro,et al.  Exploiting synergies of mobile mapping sensors and deep learning for traffic sign recognition systems , 2017, Expert Syst. Appl..

[43]  Steven Lake Waslander,et al.  Joint 3D Proposal Generation and Object Detection from View Aggregation , 2017, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[44]  Yin Zhou,et al.  VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[45]  Kaiming He,et al.  Focal Loss for Dense Object Detection , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[46]  M Marvin Raaijmakers,et al.  Towards environment perception for highly automated driving:with a case study on roundabouts , 2017 .

[47]  Enoc Sanz-Ablanedo,et al.  Automatic Detection and Classification of Pole-Like Objects for Urban Cartography Using Mobile Laser Scanning Data , 2017, Sensors.

[48]  Yiping Chen,et al.  Traffic Sign Occlusion Detection Using Mobile Laser Scanning Point Clouds , 2017, IEEE Transactions on Intelligent Transportation Systems.

[49]  Jing Huang,et al.  Point cloud labeling using 3D Convolutional Neural Network , 2016, 2016 23rd International Conference on Pattern Recognition (ICPR).

[50]  Sergio Guadarrama,et al.  Speed/Accuracy Trade-Offs for Modern Convolutional Object Detectors , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[51]  Bo Li,et al.  3D fully convolutional network for vehicle detection in point cloud , 2016, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[52]  Dushyant Rao,et al.  Vote3Deep: Fast object detection in 3D point clouds using efficient convolutional neural networks , 2016, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[53]  Johann Marius Zöllner,et al.  DeepTLR: A single deep convolutional network for detection and classification of traffic lights , 2016, 2016 IEEE Intelligent Vehicles Symposium (IV).

[54]  Zhaohui Wu,et al.  Weakly Supervised Metric Learning for Traffic Sign Recognition in a LIDAR-Equipped Vehicle , 2016, IEEE Transactions on Intelligent Transportation Systems.

[55]  Lin Li,et al.  A method based on an adaptive radius cylinder model for detecting pole-like objects in mobile laser scanning data , 2016 .

[56]  Cheng Wang,et al.  Bag-of-visual-phrases and hierarchical deep models for traffic sign detection and recognition in mobile laser scanning data , 2016 .

[57]  Mohan M. Trivedi,et al.  Vision for Looking at Traffic Lights: Issues, Survey, and Perspectives , 2016, IEEE Transactions on Intelligent Transportation Systems.

[58]  Jouko Lampinen,et al.  Object Classification and Recognition From Mobile Laser Scanning Point Clouds in a Road Environment , 2016, IEEE Transactions on Geoscience and Remote Sensing.

[59]  Wei Liu,et al.  SSD: Single Shot MultiBox Detector , 2015, ECCV.

[60]  Jing Huang,et al.  Pole-like object detection and classification from urban point clouds , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[61]  Jun Yu,et al.  Semiautomated Extraction of Street Light Poles From Mobile LiDAR Point-Clouds , 2015, IEEE Transactions on Geoscience and Remote Sensing.

[62]  Cheng Wang,et al.  Automated Extraction of Urban Road Facilities Using Mobile Laser Scanning Data , 2015, IEEE Transactions on Intelligent Transportation Systems.

[63]  Jian Sun,et al.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[64]  Zhidong Deng,et al.  LIDAR and vision-based real-time traffic sign detection and recognition algorithm for intelligent vehicle , 2014, 17th International IEEE Conference on Intelligent Transportation Systems (ITSC).

[65]  Klaus C. J. Dietmayer,et al.  Robust and real-time multi-cue map verification for the road ahead , 2014, 17th International IEEE Conference on Intelligent Transportation Systems (ITSC).

[66]  Klaus C. J. Dietmayer,et al.  Towards autonomous self-assessment of digital maps , 2014, 2014 IEEE Intelligent Vehicles Symposium Proceedings.

[67]  Jay A. Farrell,et al.  Traffic sign detection, state estimation, and identification using onboard sensors , 2013, 16th International IEEE Conference on Intelligent Transportation Systems (ITSC 2013).

[68]  Clément Zinoune,et al.  A sequential test for autonomous localisation of map errors for driving assistance systems , 2012, 2012 15th International IEEE Conference on Intelligent Transportation Systems.

[69]  Clément Zinoune,et al.  Detection of missing roundabouts in maps for Driving Assistance Systems , 2012, 2012 IEEE Intelligent Vehicles Symposium.

[70]  Chris Urmson,et al.  Traffic light mapping and detection , 2011, 2011 IEEE International Conference on Robotics and Automation.

[71]  Vladimir G. Kim,et al.  Shape-based recognition of 3D point clouds in urban environments , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[72]  B. Thomas,et al.  The ActMAP - FeedMAP framework A basis for in-vehicle ADAS application improvement , 2008, 2008 IEEE Intelligent Vehicles Symposium.

[73]  J. Seewig,et al.  Insufficiency-Driven DNN Error Detection in the Context of SOTIF on Traffic Sign Recognition Use Case , 2023, IEEE Open Journal of Intelligent Transportation Systems.

[74]  M. Gharib,et al.  Robust Traffic Sign Recognition Against Camera Failures , 2022, IEEE Open Journal of Intelligent Transportation Systems.

[75]  Li Li,et al.  Extraction of Street Pole-Like Objects Based on Plane Filtering From Mobile LiDAR Data , 2021, IEEE Trans. Geosci. Remote. Sens..

[76]  Sheng Xu,et al.  Recognizing Street Lighting Poles From Mobile LiDAR Data , 2017, IEEE Transactions on Geoscience and Remote Sensing.

[77]  Chenglu Wen,et al.  Spatial-Related Traffic Sign Inspection for Inventory Purposes Using Mobile Laser Scanning Data , 2016, IEEE Transactions on Intelligent Transportation Systems.

[78]  Pedro Arias,et al.  Automatic Segmentation and Shape-Based Classification of Retro-Reflective Traffic Signs from Mobile LiDAR Data , 2016, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[79]  Frederick R. Forst,et al.  On robust estimation of the location parameter , 1980 .