A Review of Visual-LiDAR Fusion based Simultaneous Localization and Mapping

Autonomous navigation requires both a precise and robust mapping and localization solution. In this context, Simultaneous Localization and Mapping (SLAM) is a very well-suited solution. SLAM is used for many applications including mobile robotics, self-driving cars, unmanned aerial vehicles, or autonomous underwater vehicles. In these domains, both visual and visual-IMU SLAM are well studied, and improvements are regularly proposed in the literature. However, LiDAR-SLAM techniques seem to be relatively the same as ten or twenty years ago. Moreover, few research works focus on vision-LiDAR approaches, whereas such a fusion would have many advantages. Indeed, hybridized solutions offer improvements in the performance of SLAM, especially with respect to aggressive motion, lack of light, or lack of visual features. This study provides a comprehensive survey on visual-LiDAR SLAM. After a summary of the basic idea of SLAM and its implementation, we give a complete review of the state-of-the-art of SLAM research, focusing on solutions using vision, LiDAR, and a sensor fusion of both modalities.

[1]  Daniel Cremers,et al.  Dense visual SLAM for RGB-D cameras , 2013, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[2]  Martin Lauer,et al.  LIMO: Lidar-Monocular Visual Odometry , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[3]  Paul Newman,et al.  Precise Ego-Motion Estimation with Millimeter-Wave Radar Under Diverse and Challenging Conditions , 2018, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[4]  Olivier Stasse,et al.  MonoSLAM: Real-Time Single Camera SLAM , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[5]  Paul H. J. Kelly,et al.  Dense planar SLAM , 2014, 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[6]  Conor Ryan,et al.  Deep Learning for Visual Navigation of Unmanned Ground Vehicles : A review , 2018, 2018 29th Irish Signals and Systems Conference (ISSC).

[7]  Damien Vivet,et al.  PAVO: a Parallax based Bi-Monocular VO Approach For Autonomous Navigation In Various Environments , 2019 .

[8]  Bart van Arem,et al.  Application of Driverless Electric Automated Shuttles for Public Transport in Villages: The Case of Appelscha , 2018, World Electric Vehicle Journal.

[9]  Wolfram Burgard,et al.  A Tutorial on Graph-Based SLAM , 2010, IEEE Intelligent Transportation Systems Magazine.

[10]  Wolfgang Hess,et al.  Real-time loop closure in 2D LIDAR SLAM , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[11]  Yongsheng Ou,et al.  SLAM of Robot based on the Fusion of Vision and LIDAR , 2018, 2018 IEEE International Conference on Cyborg and Bionic Systems (CBS).

[12]  J. M. M. Montiel,et al.  ORB-SLAM: A Versatile and Accurate Monocular SLAM System , 2015, IEEE Transactions on Robotics.

[13]  Andrew W. Fitzgibbon,et al.  KinectFusion: Real-time dense surface mapping and tracking , 2011, 2011 10th IEEE International Symposium on Mixed and Augmented Reality.

[14]  Ji Zhang,et al.  Visual-lidar odometry and mapping: low-drift, robust, and fast , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[15]  Stefan Leutenegger,et al.  CodeSLAM - Learning a Compact, Optimisable Representation for Dense Visual SLAM , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[16]  Wolfram Burgard,et al.  Efficient Sparse Pose Adjustment for 2D mapping , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[17]  Gert F. Trommer,et al.  Laser-aided navigation with loop closure capabilities for Micro Aerial Vehicles in indoor and urban environments , 2014, 2014 IEEE/ION Position, Location and Navigation Symposium - PLANS 2014.

[18]  Thierry Peynot,et al.  Reliable automatic camera-laser calibration , 2010, ICRA 2010.

[19]  Wolfram Burgard,et al.  Improved Techniques for Grid Mapping With Rao-Blackwellized Particle Filters , 2007, IEEE Transactions on Robotics.

[20]  Hugh F. Durrant-Whyte,et al.  Simultaneous localization and mapping: part I , 2006, IEEE Robotics & Automation Magazine.

[21]  Zhengyou Zhang,et al.  Microsoft Kinect Sensor and Its Effect , 2012, IEEE Multim..

[22]  Guolai Jiang,et al.  A Simultaneous Localization and Mapping (SLAM) Framework for 2.5D Map Building Based on Low-Cost LiDAR and Vision Fusion , 2019, Applied Sciences.

[23]  Jean-Paul Laumond,et al.  Position referencing and consistent world modeling for mobile robots , 1985, Proceedings. 1985 IEEE International Conference on Robotics and Automation.

[24]  Ryad Benosman,et al.  Simultaneous Mosaicing and Tracking with an Event Camera , 2014, BMVC.

[25]  Catherine M. Burns,et al.  Autonomous Driving in the Real World: Experiences with Tesla Autopilot and Summon , 2016, AutomotiveUI.

[26]  José Ruíz Ascencio,et al.  Visual simultaneous localization and mapping: a survey , 2012, Artificial Intelligence Review.

[27]  Dieter Fox,et al.  RGB-D mapping: Using Kinect-style depth cameras for dense 3D modeling of indoor environments , 2012, Int. J. Robotics Res..

[28]  Paul Newman,et al.  FAB-MAP: Probabilistic Localization and Mapping in the Space of Appearance , 2008, Int. J. Robotics Res..

[29]  Daniel Cremers,et al.  Event-based 3D SLAM with a depth-augmented dynamic vision sensor , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[30]  Christoph Gustav Keller,et al.  Landmark based radar SLAM using graph optimization , 2016, 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC).

[31]  Luis Miguel Bergasa,et al.  A Multi-Sensorial Simultaneous Localization and Mapping (SLAM) System for Low-Cost Micro Aerial Vehicles in GPS-Denied Environments , 2017, Sensors.

[32]  Federico Tombari,et al.  CNN-SLAM: Real-Time Dense Monocular SLAM with Learned Depth Prediction , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[33]  Roland Siegwart,et al.  Two different tools for three-dimensional mapping: DE-based scan matching and feature-based loop detection , 2013, Robotica.

[34]  Hauke Strasdat,et al.  Visual SLAM: Why filter? , 2012, Image Vis. Comput..

[35]  Lee Gomes,et al.  When will Google's self-driving car really be ready? It depends on where you live and what you mean by "ready" [News] , 2016 .

[36]  Sei Ikeda,et al.  Visual SLAM algorithms: a survey from 2010 to 2016 , 2017, IPSJ Transactions on Computer Vision and Applications.

[37]  Yuan Zhou,et al.  Research on active SLAM with fusion of monocular vision and laser range data , 2010, 2010 8th World Congress on Intelligent Control and Automation.

[38]  Kihong Park,et al.  High-Precision Depth Estimation Using Uncalibrated LiDAR and Stereo Fusion , 2020, IEEE Transactions on Intelligent Transportation Systems.

[39]  Emanuele Menegatti,et al.  A portable three-dimensional LIDAR-based system for long-term and wide-area people behavior measurement , 2019, International Journal of Advanced Robotic Systems.

[40]  Enhai Liu,et al.  Scale Estimation and Correction of the Monocular Simultaneous Localization and Mapping (SLAM) Based on Fusion of 1D Laser Range Finder and Vision Data , 2018, Sensors.

[41]  Davide Scaramuzza,et al.  SVO: Fast semi-direct monocular visual odometry , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[42]  Roland Chapuis,et al.  Localization and Mapping Using Only a Rotating FMCW Radar Sensor , 2013, Sensors.

[43]  Daniel Cremers,et al.  Direct Sparse Odometry , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[44]  Lindsay Kleeman,et al.  Fast Laser Scan Matching using Polar Coordinates , 2007, Int. J. Robotics Res..

[45]  Yunhui Liu,et al.  Visual laser-SLAM in large-scale indoor environments , 2016, 2016 IEEE International Conference on Robotics and Biomimetics (ROBIO).

[46]  Ji Zhang,et al.  Low-drift and real-time lidar odometry and mapping , 2017, Auton. Robots.

[47]  Wolfram Burgard,et al.  Improving Grid-based SLAM with Rao-Blackwellized Particle Filters by Adaptive Proposals and Selective Resampling , 2005, Proceedings of the 2005 IEEE International Conference on Robotics and Automation.

[48]  Chih-Chung Chou,et al.  A Tight Coupling of Vision-Lidar Measurements for an Effective Odometry , 2019, 2019 IEEE Intelligent Vehicles Symposium (IV).

[49]  Evangelos E. Milios,et al.  Globally Consistent Range Scan Alignment for Environment Mapping , 1997, Auton. Robots.

[50]  Udo Frese,et al.  A Discussion of Simultaneous Localization and Mapping , 2006, Auton. Robots.

[51]  Nassir Navab,et al.  When 2.5D is not enough: Simultaneous reconstruction, segmentation and recognition on dense SLAM , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[52]  Hugh F. Durrant-Whyte,et al.  Mobile robot localization by tracking geometric beacons , 1991, IEEE Trans. Robotics Autom..

[53]  Wolfram Burgard,et al.  An evaluation of the RGB-D SLAM system , 2012, 2012 IEEE International Conference on Robotics and Automation.

[54]  J DavisonAndrew,et al.  Editors Choice Article , 2012 .

[55]  G. Klein,et al.  Parallel Tracking and Mapping for Small AR Workspaces , 2007, 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality.

[56]  Andrew J. Davison,et al.  Real-time simultaneous localisation and mapping with a single camera , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[57]  Huadong Dai,et al.  Loop Detection and Correction of 3D Laser-Based SLAM with Visual Information , 2018, CASA 2018.

[58]  Cyrill Stachniss,et al.  Accurate Direct Visual-Laser Odometry with Explicit Occlusion Handling and Plane Detection , 2019, 2019 International Conference on Robotics and Automation (ICRA).

[59]  Juan D. Tardós,et al.  ORB-SLAM2: An Open-Source SLAM System for Monocular, Stereo, and RGB-D Cameras , 2016, IEEE Transactions on Robotics.

[60]  Ayoung Kim,et al.  Direct Visual SLAM Using Sparse Depth for Camera-LiDAR System , 2018, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[61]  Stefan Leutenegger,et al.  Real-Time 3D Reconstruction and 6-DoF Tracking with an Event Camera , 2016, ECCV.

[62]  Ahmet M. Kondoz,et al.  Fusion of LiDAR and Camera Sensor Data for Environment Sensing in Driverless Vehicles , 2017, ArXiv.

[63]  David W. Murray,et al.  Video-rate localization in multiple maps for wearable augmented reality , 2008, 2008 12th IEEE International Symposium on Wearable Computers.

[64]  Ian D. Reid,et al.  A Constant-Time Efficient Stereo SLAM System , 2009, BMVC.

[65]  Silvio Savarese,et al.  Visually bootstrapped generalized ICP , 2011, 2011 IEEE International Conference on Robotics and Automation.

[66]  Sebastian Scherer,et al.  River mapping from a flying robot: state estimation, river detection, and obstacle mapping , 2012, Auton. Robots.

[67]  Daniel Cremers,et al.  LSD-SLAM: Large-Scale Direct Monocular SLAM , 2014, ECCV.

[68]  Andrew J. Davison,et al.  DTAM: Dense tracking and mapping in real-time , 2011, 2011 International Conference on Computer Vision.

[69]  Andreas Nüchter,et al.  Automatic Appearance-Based Loop Detection from 3 D Laser Data Using the Normal Distributions Transform , 2009 .

[70]  Damien Vivet Extracting Proprioceptive Information By Analyzing Rotating Range Sensors Induced Distortion , 2019, 2019 27th European Signal Processing Conference (EUSIPCO).

[71]  Daniel Cremers,et al.  Stereo DSO: Large-Scale Direct Sparse Visual Odometry with Stereo Cameras , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[72]  Shahram Izadi,et al.  MonoFusion: Real-time 3D reconstruction of small scenes with a single web camera , 2013, 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[73]  Jason Geng,et al.  Structured-light 3D surface imaging: a tutorial , 2011 .

[74]  Rongbing Li,et al.  LIDAR/MEMS IMU integrated navigation (SLAM) method for a small UAV in indoor environments , 2014, 2014 DGON Inertial Sensors and Systems (ISS).

[75]  Joachim Hertzberg,et al.  6D SLAM—3D mapping outdoor environments , 2007, J. Field Robotics.

[76]  Paul J. Besl,et al.  A Method for Registration of 3-D Shapes , 1992, IEEE Trans. Pattern Anal. Mach. Intell..