Multiview CNN Model for Sensor Fusion Based Vehicle Detection

The self-diving vehicle is a critical revolution in the automobile industry, transportation and people’s daily life. Generally, self-driving vehicles utilize the LIDAR, IMU, radar along with traditional optical cameras for sensing surrounding environment. Therefore, on road object detection can be improved by utilizing multiview of LIDAR and camera instead of single sensor. In this paper, we propose a sensor data fusion-based multiview CNN model for on road vehicle detection. We first up-sample the sparse LIDAR point cloud as gray images, and then combine the up-sampled gray image (F channel) with the camera image (RGB channels) as four-channel fusion data to train a CNN model. To reduce the redundant data, we tested the performance of CNN models by different fusion schemes. In this way, we can reduce the consumption of the CNN model in both training and on board testing for vehicular environment. Our final fusion model (RBF) can reach an average accuracy of 82% on the real trace dataset of KITTI.

[1]  Ling Shao,et al.  Color object recognition via cross-domain learning on RGB-D images , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[2]  Ali Farhadi,et al.  YOLO9000: Better, Faster, Stronger , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Alois Knoll,et al.  Vehicle detection based on LiDAR and camera fusion , 2014, 17th International IEEE Conference on Intelligent Transportation Systems (ITSC).

[4]  Alade O. Tokuta,et al.  Fusion of color images and LiDAR data for lane classification , 2015, SIGSPATIAL/GIS.

[5]  Pietro Perona,et al.  Fast Feature Pyramids for Object Detection , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[6]  Ali Farhadi,et al.  You Only Look Once: Unified, Real-Time Object Detection , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Paulo Peixoto,et al.  Real-Time Deep ConvNet-Based Vehicle Detection Using 3D-LIDAR Reflection Intensity Data , 2017, ROBOT.

[8]  Trevor Darrell,et al.  Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation , 2013, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[9]  James J. Little,et al.  The Raincouver Scene Parsing Benchmark for Self-Driving in Adverse Weather and at Night , 2017, IEEE Robotics and Automation Letters.

[10]  Youjie Zhou,et al.  Co-Interest Person Detection from Multiple Wearable Camera Videos , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[11]  Xuelong Li,et al.  A Multiview-Based Parameter Free Framework for Group Detection , 2017, AAAI.

[12]  Cristiano Premebida,et al.  Pedestrian detection combining RGB and dense LIDAR data , 2014, 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[13]  Tian Xia,et al.  Vehicle Detection from 3D Lidar Using Fully Convolutional Network , 2016, Robotics: Science and Systems.

[14]  Andreas Geiger,et al.  Are we ready for autonomous driving? The KITTI vision benchmark suite , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[15]  Tat-Seng Chua,et al.  Unifying Virtual and Physical Worlds , 2017, ACM Trans. Inf. Syst..

[16]  Michael Ying Yang,et al.  Real-time RGB-D based template matching pedestrian detection , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[17]  Michael L. Nelson,et al.  Comparing the Archival Rate of Arabic, English, Danish, and Korean Language Web Pages , 2017, ACM Trans. Inf. Syst..

[18]  Yu Liu,et al.  Multiwave: A novel vehicle steering pattern detection method based on smartphones , 2016, 2016 IEEE International Conference on Communications (ICC).

[19]  Namil Kim,et al.  Multispectral pedestrian detection: Benchmark dataset and baseline , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Ross B. Girshick,et al.  Fast R-CNN , 2015, 1504.08083.

[21]  Jason Borenstein,et al.  Self-Driving Cars: Ethical Responsibilities of Design Engineers , 2017, IEEE Technology and Society Magazine.

[22]  Xinming Huang,et al.  End-to-end learning for lane keeping of self-driving cars , 2017, 2017 IEEE Intelligent Vehicles Symposium (IV).

[23]  Yu Liu,et al.  A cGANs-Based Scene Reconstruction Model Using Lidar Point Cloud , 2017, 2017 IEEE International Symposium on Parallel and Distributed Processing with Applications and 2017 IEEE International Conference on Ubiquitous Computing and Communications (ISPA/IUCC).

[24]  Shiyu Song,et al.  Joint SFM and detection cues for monocular 3D localization in road scenes , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Nanning Zheng,et al.  A vision-centered multi-sensor fusing approach to self-localization and obstacle perception for robotic cars , 2017, Frontiers of Information Technology & Electronic Engineering.

[26]  Leonidas J. Guibas,et al.  Volumetric and Multi-view CNNs for Object Classification on 3D Data , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  Jian Sun,et al.  Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition , 2015, IEEE Trans. Pattern Anal. Mach. Intell..

[28]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[29]  Rongrong Ji,et al.  Towards 3D object detection with bimodal deep Boltzmann machines over RGBD imagery , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Tat-Seng Chua,et al.  Micro Tells Macro: Predicting the Popularity of Micro-Videos via a Transductive Model , 2016, ACM Multimedia.

[31]  Paulo Peixoto,et al.  DepthCN: Vehicle detection using 3D-LIDAR and ConvNet , 2017, 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC).