A Cascaded LiDAR-Camera Fusion Network for Road Detection

Most of the existing road detection methods are either single-modal based, e.g., based on LiDAR or camera, or multi-modal based with LiDAR-camera fusion. The algorithms are designed for a specific data type, and cannot cope with input data changes. In addition, the LiDAR-camera based methods can only work in day time with enough light. In this paper, we develop a novel LiDAR-camera fusion strategy, which combines the LiDAR point clouds and the camera images in a cascaded way. The proposed network has two working modes, the single-modal mode with LiDAR point clouds only and the multimodal mode with both LiDAR and camera data, so it can be used in all day scenes. The whole network consists of three parts: 1) LiDAR segmentation module, which segments road points in the LiDAR’s imagery view. 2) Sparse-to-dense module, which upsamples the sparse LiDAR feature maps to dense road detection results. 3) LiDAR-camera fusion module, which fuses the dense LiDAR feature maps with the dense camera images to obtain accurate road estimations. Experiments on the KITTI-Road dataset show that the proposed cascaded LiDAR-camera fusion network can obtain very competitive road detection performance, with a MaxF value of 96.38%, and achieve the state-of-the-art in the single-modal mode among all LiDAR-only methods.

[1]  Rui Fan,et al.  SNE-RoadSeg: Incorporating Surface Normal Information into Semantic Segmentation for Accurate Freespace Detection , 2020, ECCV.

[2]  Michael Felsberg,et al.  Confidence Propagation through CNNs for Guided Sparse Depth Regression , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[3]  Shuo Gu,et al.  Two-View Fusion based Convolutional Neural Network for Urban Road Detection , 2019, 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[4]  Jee-Young Sun,et al.  Reverse and Boundary Attention Network for Road Segmentation , 2019, 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW).

[5]  Hui Kong,et al.  Road Detection through CRF based LiDAR-Camera Fusion , 2019, 2019 International Conference on Robotics and Automation (ICRA).

[6]  Zhe Chen,et al.  Progressive LiDAR adaptation for road detection , 2019, IEEE/CAA Journal of Automatica Sinica.

[7]  Sertac Karaman,et al.  Self-Supervised Sparse-to-Dense: Self-Supervised Depth Completion from LiDAR and Monocular Camera , 2018, 2019 International Conference on Robotics and Automation (ICRA).

[8]  Fawzi Nashashibi,et al.  Sparse and Dense Data with CNNs: Depth Completion and Semantic Segmentation , 2018, 2018 International Conference on 3D Vision (3DV).

[9]  Hongdong Li,et al.  Semisupervised and Weakly Supervised Road Detection Based on Generative Adversarial Networks , 2018, IEEE Signal Processing Letters.

[10]  Liang Xiao,et al.  Hybrid conditional random field based camera-LIDAR fusion for road detection , 2017, Inf. Sci..

[11]  Roberto Cipolla,et al.  MultiNet: Real-time Joint Semantic Reasoning for Autonomous Driving , 2016, 2018 IEEE Intelligent Vehicles Symposium (IV).

[12]  Zhe Chen,et al.  RBNet: A Deep Neural Network for Unified Road and Road Boundary Detection , 2017, ICONIP.

[13]  Lennart Svensson,et al.  Fast LIDAR-based road detection using fully convolutional neural networks , 2017, 2017 IEEE Intelligent Vehicles Symposium (IV).

[14]  Wolfram Burgard,et al.  Efficient deep models for monocular road segmentation , 2016, 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[15]  Zsolt Kira,et al.  Fusing LIDAR and images for pedestrian detection using convolutional neural networks , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[16]  Vladlen Koltun,et al.  Multi-Scale Context Aggregation by Dilated Convolutions , 2015, ICLR.

[17]  Wolfram Burgard,et al.  Multimodal deep learning for robust RGB-D object recognition , 2015, 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[18]  Liang Xiao,et al.  CRF based road detection with multi-sensor fusion , 2015, 2015 IEEE Intelligent Vehicles Symposium (IV).

[19]  Jannik Fritsch,et al.  A new performance measure and evaluation benchmark for road detection algorithms , 2013, 16th International IEEE Conference on Intelligent Transportation Systems (ITSC 2013).