Incorporating learnt local and global embeddings into monocular visual SLAM

Traditional approaches for Visual Simultaneous Localization and Mapping (VSLAM) rely on low-level vision information for state estimation, such as handcrafted local features or the image gradient. While significant progress has been made through this track, under more challenging configuration for monocular VSLAM, e.g., varying illumination, the performance of state-of-the-art systems generally degrades. As a consequence, robustness and accuracy for monocular VSLAM are still widely concerned. This paper presents a monocular VSLAM system that fully exploits learnt features for better state estimation. The proposed system leverages both learnt local features and global embeddings at different modules of the system: direct camera pose This work was supported by the National Natural Science Foundation of China, under grant No. U1713211, Collaborative Research Fund by Research Grants Council Hong Kong, under Project No. C4063-18G, and HKUST-SJTU Joint Research Collaboration Fund, under project SJTU20EG03, awarded to Prof. Ming Liu. Ming Liu is the corresponding author. · Huaiyang Huang E-mail: hhuangat@connect.ust.hk Haoyang Ye E-mail: hy.ye@connect.ust.hk Lujia Wang E-mail: eewanglj@ust.hk Ming Liu (corresponding author) E-mail: eelium@ust.hk Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology, Hong Kong, SAR, China · Yuxiang Sun E-mail: yx.sun@polyu.edu.hk, sun.yuxiang@outlook.com Department of Mechanical Engineering, The Hong Kong Polytechnic University, Hung Hom Hong Kong, SAR, China estimation, inter-frame feature association, and loop closure detection. With a probabilistic explanation of keypoint prediction, we formulate the camera pose tracking in a direct manner and parameterize local features with uncertainty taken into account. To alleviate the quantization effect, we adapt the mapping module to generate 3D landmarks better to guarantee the system’s robustness. Detecting temporal loop closure via deep global embeddings further improves the robustness and accuracy of the proposed system. The proposed system is extensively evaluated on public datasets (Tsukuba, EuRoC, and KITTI), and compared against the state-of-theart methods. The competitive performance of camera pose estimation confirms the effectiveness of our method.