ImLoveNet: Misaligned Image-supported Registration Network for Low-overlap Point Cloud Pairs

Low-overlap regions between paired point clouds make the captured features very low-confidence, leading cutting edge models to point cloud registration with poor quality. Beyond the traditional wisdom, we raise an intriguing question: Is it possible to exploit an intermediate yet misaligned image between two low-overlap point clouds to enhance the performance of cutting-edge registration models? To answer it, we propose a misaligned image supported registration network for low-overlap point cloud pairs, dubbed ImLoveNet. ImLoveNet first learns triple deep features across different modalities and then exports these features to a two-stage classifier, for progressively obtaining the high-confidence overlap region between the two point clouds. Therefore, soft correspondences are well established on the predicted overlap region, resulting in accurate rigid transformations for registration. ImLoveNet is simple to implement yet effective, since 1) the misaligned image provides clearer overlap information for the two low-overlap point clouds to better locate overlap parts; 2) it contains certain geometry knowledge to extract better deep features; and 3) it does not require the extrinsic parameters of the imaging device with respect to the reference frame of the 3D point cloud. Extensive qualitative and quantitative evaluations on different kinds of benchmarks demonstrate the effectiveness and superiority of our ImLoveNet over state-of-the-art approaches.

[1]  Zhen Dong,et al.  You Only Hypothesize Once: Point Cloud Registration with Rotation-equivariant Descriptors , 2021, ACM Multimedia.

[2]  Bichen Wu,et al.  Image2Point: 3D Point-Cloud Understanding with 2D Image Pretrained Models , 2021, ECCV.

[3]  Daniel Cohen-Or,et al.  Consistent Two-Flow Network for Tele-Registration of Point Clouds , 2021, IEEE Transactions on Visualization and Computer Graphics.

[4]  L. Gool,et al.  Learning to Relate Depth and Semantics for Unsupervised Domain Adaptation , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Saining Xie,et al.  Pri3D: Can 3D Priors Help 2D Representation Learning? , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[6]  Jiaxin Li,et al.  DeepI2P: Image-to-Point Cloud Registration via Deep Classification , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Chi-Wing Fu,et al.  3D-to-2D Distillation for Indoor Scene Parsing , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Chiew-Lan Tai,et al.  PointDSC: Robust Point Cloud Registration using Deep Spatial Consistency , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Bing Zeng,et al.  OMNet: Learning Overlapping Mask for Partial-to-Partial Point Cloud Registration , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[10]  K. Schindler,et al.  PREDATOR: Registration of 3D Point Clouds with Low Overlap , 2020, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Zhe L. Lin,et al.  SDC-Depth: Semantic Divide-and-Conquer Network for Monocular Depth Estimation , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Jian Zhang,et al.  Feature-Metric Registration: A Fast Semi-Supervised Approach for Robust Point Cloud Registration Without Correspondences , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Vladlen Koltun,et al.  Deep Global Registration , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Zi Jian Yew,et al.  RPM-Net: Robust Point Matching Using Learned Features , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Long Quan,et al.  D3Feat: Joint Learning of Dense Detection and Description of 3D Local Features , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  J. D. Wegner,et al.  Learning Multiview 3D Point Cloud Registration , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Yue Wang,et al.  PRNet: Self-Supervised Learning for Partial-to-Partial Registration , 2019, NeurIPS.

[18]  Matthias Nießner,et al.  End-to-End CAD Model Retrieval and 9DoF Alignment in 3D Scans , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[19]  Yue Wang,et al.  Deep Closest Point: Learning Representations for Point Cloud Registration , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[20]  R. Chellappa,et al.  3DRegNet: A Deep Neural Network for 3D Point Registration , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Yasuhiro Aoki,et al.  PointNetLK: Robust & Efficient Point Cloud Registration Using PointNet , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Andreas Wieser,et al.  The Perfect Match: 3D Point Cloud Matching With Smoothed Densities , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Slobodan Ilic,et al.  PPF-FoldNet: Unsupervised Learning of Rotation Invariant 3D Local Descriptors , 2018, ECCV.

[24]  Renjie Liao,et al.  GeoNet: Geometric Neural Network for Joint Depth and Surface Normal Estimation , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[25]  Slobodan Ilic,et al.  PPFNet: Global Context Aware Local Features for Robust 3D Point Matching , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[26]  Yue Wang,et al.  Dynamic Graph CNN for Learning on Point Clouds , 2018, ACM Trans. Graph..

[27]  Dieter Fox,et al.  PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes , 2017, Robotics: Science and Systems.

[28]  Leonidas J. Guibas,et al.  PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space , 2017, NIPS.

[29]  Xiaogang Wang,et al.  Pyramid Scene Parsing Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Leonidas J. Guibas,et al.  PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[31]  Matthias Nießner,et al.  BundleFusion , 2016, TOGS.

[32]  Matthias Nießner,et al.  3DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructions , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[33]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[34]  Mohammed Bennamoun,et al.  Performance Evaluation of 3D Local Feature Descriptors , 2014, ACCV.

[35]  Andreas Geiger,et al.  Vision meets robotics: The KITTI dataset , 2013, Int. J. Robotics Res..

[36]  Bertram Drost,et al.  3D Object Detection and Localization Using Multimodal Point Pair Features , 2012, 2012 Second International Conference on 3D Imaging, Modeling, Processing, Visualization & Transmission.

[37]  Federico Tombari,et al.  A combined texture-shape descriptor for enhanced 3D feature matching , 2011, 2011 18th IEEE International Conference on Image Processing.

[38]  Nico Blodow,et al.  Fast Point Feature Histograms (FPFH) for 3D registration , 2009, 2009 IEEE International Conference on Robotics and Automation.