Loop closure detection using supervised and unsupervised deep neural networks for monocular SLAM systems

Abstract The detection of true loop closure in Visual Simultaneous Localization And Mapping (vSLAM) can help in many ways, it helps in re-localization, improves the accuracy of the map, and helps in registration algorithms to obtain more accurate and consistent results. The loop closure detection is affected by many parameters, including illumination conditions, seasons, different viewpoints and mobile objects. This paper proposes a novel approach based on super dictionary different from traditional BoW dictionary that uses more advanced and more abstract features of deep learning. The proposed approach does not need to generate vocabulary, which makes it memory efficient and instead it stores exact features, which are small in number and hold very less amount of memory as compared to traditional BoW approach in which each frame holds the same amount of memory as the number of words in the vocabulary. Two deep neural networks are used together to speed up the loop closure detection and to ignore the effect of mobile objects on loop closure detection. We have compared the results with most popular Bag of Words methods DBoW2 and DBoW3, and state-of-the-art iBoW-LCD using five publicly available datasets, and the results show that the proposed method robustly performs loop closure detection and is eight times faster than the state-of-the-art approaches of a similar kind.

[1]  Ali Farhadi,et al.  YOLO9000: Better, Faster, Stronger , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  Yong Liu,et al.  Robust visual SLAM with point and line features , 2017, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[3]  Tomohiro Shibata,et al.  High performance loop closure detection using bag of word pairs , 2016, Robotics Auton. Syst..

[4]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[5]  Yury Malkov,et al.  Revisiting the Inverted Indices for Billion-Scale Approximate Nearest Neighbors , 2018, ECCV.

[6]  Wolfram Burgard,et al.  Robust Visual Localization Across Seasons , 2018, IEEE Transactions on Robotics.

[7]  Dorian Gálvez-López,et al.  Bags of Binary Words for Fast Place Recognition in Image Sequences , 2012, IEEE Transactions on Robotics.

[8]  Tao Zhang,et al.  Unsupervised learning to detect loops using deep neural networks for visual SLAM system , 2017, Auton. Robots.

[9]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[10]  Yan Su,et al.  Graph-Based Place Recognition in Image Sequences with CNN Features , 2018, Journal of Intelligent & Robotic Systems.

[11]  Hanbin Xi,et al.  Image Classification Based on Histogram Intersection Kernel , 2015 .

[12]  Clement Chung,et al.  Implementation of an integrated computerized prescriber order‐entry system for chemotherapy in a multisite safety‐net health system , 2018, American journal of health-system pharmacy : AJHP : official journal of the American Society of Health-System Pharmacists.

[13]  Andrey A. Filchenkov,et al.  Application of deep learning to the problem of vehicle detection in UAV images , 2016, 2016 XIX IEEE International Conference on Soft Computing and Measurements (SCM).

[14]  Tao Zhang,et al.  Loop closure detection for visual SLAM systems using deep neural networks , 2015, 2015 34th Chinese Control Conference (CCC).

[15]  Gary R. Bradski,et al.  ORB: An efficient alternative to SIFT or SURF , 2011, 2011 International Conference on Computer Vision.

[16]  Tao Zhang,et al.  Semi-direct monocular visual and visual-inertial SLAM with loop closure detection , 2019, Robotics Auton. Syst..

[17]  Muhamad Risqi U. Saputra,et al.  Visual SLAM and Structure from Motion in Dynamic Environments , 2018, ACM Comput. Surv..

[18]  Youcef Mezouar,et al.  Multi-resolution map building and loop closure with omnidirectional images , 2017, Auton. Robots.

[19]  Juan D. Tardós,et al.  ORB-SLAM2: An Open-Source SLAM System for Monocular, Stereo, and RGB-D Cameras , 2016, IEEE Transactions on Robotics.

[20]  Alberto Ortiz,et al.  iBoW-LCD: An Appearance-Based Loop-Closure Detection Approach Using Incremental Bags of Binary Words , 2018, IEEE Robotics and Automation Letters.

[21]  Charles Richter,et al.  Safe Visual Navigation via Deep Learning and Novelty Detection , 2017, Robotics: Science and Systems.

[22]  Roland Siegwart,et al.  Robust Visual Place Recognition with Graph Kernels , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Andreas Geiger,et al.  Are we ready for autonomous driving? The KITTI vision benchmark suite , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[24]  Yong Liu,et al.  Graph Regularized Auto-Encoders for Image Representation. , 2017, IEEE transactions on image processing : a publication of the IEEE Signal Processing Society.

[25]  Peter Auer,et al.  Generic object recognition with boosting , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[26]  Hexin Zhang,et al.  A Loop Closure Detection Algorithm in Dynamic Scene , 2018, DEStech Transactions on Computer Science and Engineering.

[27]  Yan Su,et al.  Loop closure detection for visual SLAM systems using convolutional neural network , 2017, 2017 23rd International Conference on Automation and Computing (ICAC).

[28]  Bernd Girod,et al.  Inverted Index Compression for Scalable Image Matching , 2010, 2010 Data Compression Conference.

[29]  Charles E. Thorpe,et al.  Simultaneous localization and mapping with detection and tracking of moving objects , 2002, Proceedings 2002 IEEE International Conference on Robotics and Automation (Cat. No.02CH37292).

[30]  Guoquan Huang,et al.  Lightweight Unsupervised Deep Loop Closure , 2018, Robotics: Science and Systems.

[31]  J. M. M. Montiel,et al.  ORB-SLAM: A Versatile and Accurate Monocular SLAM System , 2015, IEEE Transactions on Robotics.

[32]  Paolo Valigi,et al.  Robust visual semi-semantic loop closure detection by a covisibility graph and CNN features , 2017, Robotics Auton. Syst..

[33]  Hesheng Wang,et al.  Depth estimation for deformable object using a multi-layer neural network , 2017, 2017 IEEE International Conference on Real-time Computing and Robotics (RCAR).

[34]  Michael Milford,et al.  Convolutional Neural Network-based Place Recognition , 2014, ICRA 2014.

[35]  Jie Li,et al.  An Evaluation of Deep Learning in Loop Closure Detection for Visual SLAM , 2017, 2017 IEEE International Conference on Internet of Things (iThings) and IEEE Green Computing and Communications (GreenCom) and IEEE Cyber, Physical and Social Computing (CPSCom) and IEEE Smart Data (SmartData).

[36]  Jörg Stückler,et al.  Multi-resolution surfel maps for efficient dense 3D modeling and tracking , 2014, J. Vis. Commun. Image Represent..

[37]  Sergey Starkov,et al.  Moving Object Detection in Video Streams Received from a Moving Camera , 2014, AIST.

[38]  Michael Felsberg,et al.  Deep motion features for visual tracking , 2016, 2016 23rd International Conference on Pattern Recognition (ICPR).

[39]  Wolfram Burgard,et al.  3-D Mapping With an RGB-D Camera , 2014, IEEE Transactions on Robotics.

[40]  Yo-Sung Ho,et al.  Loop Closure Detection in Simultaneous Localization and Mapping Using Learning Based Local Patch Descriptor , 2018 .

[41]  Wolfram Burgard,et al.  Robust visual SLAM across seasons , 2015, 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[42]  Daniel Cremers,et al.  LDSO: Direct Sparse Odometry with Loop Closure , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[43]  Shaohua Li,et al.  Autonomous exploration of mobile robots through deep neural networks , 2017 .

[44]  Hao Qin,et al.  Loop closure detection in SLAM by combining visual CNN features and submaps , 2018, 2018 4th International Conference on Control, Automation and Robotics (ICCAR).

[45]  Hugh F. Durrant-Whyte,et al.  Simultaneous Localization, Mapping and Moving Object Tracking , 2007, Int. J. Robotics Res..

[46]  François Michaud,et al.  Online global loop closure detection for large-scale multi-session graph-based SLAM , 2014, 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[47]  Paul Newman,et al.  FAB-MAP: Probabilistic Localization and Mapping in the Space of Appearance , 2008, Int. J. Robotics Res..

[48]  Baifan Chen,et al.  Loop Closure Detection Based on Multi-Scale Deep Feature Fusion , 2019, Applied Sciences.

[49]  Jonathan Krause,et al.  3D Object Representations for Fine-Grained Categorization , 2013, 2013 IEEE International Conference on Computer Vision Workshops.