Learning Multiview 3D Point Cloud Registration

We present a novel, end-to-end learnable, multiview 3D point cloud registration algorithm. Registration of multiple scans typically follows a two-stage pipeline: the initial pairwise alignment and the globally consistent refinement. The former is often ambiguous due to the low overlap of neighboring point clouds, symmetries and repetitive scene parts. Therefore, the latter global refinement aims at establishing the cyclic consistency across multiple scans and helps in resolving the ambiguous cases. In this paper we propose, to the best of our knowledge, the first end-to-end algorithm for joint learning of both parts of this two-stage problem. Experimental evaluation on well accepted benchmark datasets shows that our approach outperforms the state-of-the-art by a significant margin, while being end-to-end trainable and computationally less costly. Moreover, we present detailed analysis and an ablation study that validate the novel components of our approach. The source code and pretrained models are publicly available under https://github.com/zgojcic/3D_multiview_reg.

[1]  P. Holland,et al.  Robust regression using iteratively reweighted least-squares , 1977 .

[2]  Robert C. Bolles,et al.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography , 1981, CACM.

[3]  Paul J. Besl,et al.  A Method for Registration of 3-D Shapes , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[4]  Andrew E. Johnson,et al.  Using Spin Images for Efficient Object Recognition in Cluttered 3D Scenes , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[5]  Martial Hebert,et al.  Fully automatic registration of multiple 3D data sets , 2003, Image Vis. Comput..

[6]  Philip H. S. Torr,et al.  The Development and Comparison of Robust Methods for Estimating the Fundamental Matrix , 1997, International Journal of Computer Vision.

[7]  Venu Madhav Govindu,et al.  Lie-algebraic averaging for globally consistent motion estimation , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[8]  Mohammed Bennamoun,et al.  Three-Dimensional Model-Based Object Recognition and Segmentation in Cluttered Scenes , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[9]  Anton van den Hengel,et al.  Thrift: Local 3D Structure Recognition , 2007, 9th Biennial Conference of the Australian Pattern Recognition Society on Digital Image Computing Techniques and Applications (DICTA 2007).

[10]  George Vosselman,et al.  An integrated approach for modelling and global registration of point clouds , 2007 .

[11]  Hongdong Li,et al.  The 3D-3D Registration Problem Revisited , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[12]  Daniel Cohen-Or,et al.  4-points congruent sets for robust pairwise surface registration , 2008, ACM Trans. Graph..

[13]  Nico Blodow,et al.  Fast Point Feature Histograms (FPFH) for 3D registration , 2009, 2009 IEEE International Conference on Robotics and Automation.

[14]  Federico Tombari,et al.  Unique Signatures of Histograms for Local Surface Description , 2010, ECCV.

[15]  Federico Tombari,et al.  Unique shape context for 3d data description , 2010, 3DOR '10.

[16]  Nassir Navab,et al.  Model globally, match locally: Efficient and robust 3D object recognition , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[17]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[18]  Andrea Torsello,et al.  Multiview registration via graph diffusion of dual quaternions , 2011, CVPR 2011.

[19]  Ira Kemelmacher-Shlizerman,et al.  Global Motion Estimation from Point Matches , 2012, 2012 Second International Conference on 3D Imaging, Modeling, Processing, Visualization & Transmission.

[20]  Andrea Fusiello,et al.  Accurate and Automatic Alignment of Range Surfaces , 2012, 2012 Second International Conference on 3D Imaging, Modeling, Processing, Visualization & Transmission.

[21]  A. Singer,et al.  Vector diffusion maps and the connection Laplacian , 2011, Communications on pure and applied mathematics.

[22]  Venu Madhav Govindu,et al.  Efficient and Robust Large-Scale Rotation Averaging , 2013, 2013 IEEE International Conference on Computer Vision.

[23]  Jan-Michael Frahm,et al.  USAC: A Universal Framework for Random Sample Consensus , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[24]  Marc Pollefeys,et al.  Automatic Registration of RGB-D Scans via Salient Directions , 2013, 2013 IEEE International Conference on Computer Vision.

[25]  Niloy J. Mitra,et al.  Super4PCS: Fast Global Pointcloud Registration via Smart Indexing , 2019 .

[26]  B. Rossi,et al.  Robust Absolute Rotation Estimation via Low-Rank and Sparse Matrix Decomposition , 2014, 2014 2nd International Conference on 3D Vision.

[27]  Andreas Krause,et al.  Advances in Neural Information Processing Systems (NIPS) , 2014 .

[28]  Konrad Schindler,et al.  Keypoint-based 4-Points Congruent Sets – Automated marker-less registration of laser scans , 2014 .

[29]  Vladlen Koltun,et al.  Robust reconstruction of indoor scenes , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Ping Tan,et al.  Global Structure-from-Motion by Similarity Averaging , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[31]  Johan Thunberg,et al.  A solution for multi-alignment by transformation synchronisation , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[33]  Slobodan Ilic,et al.  Point Pair Features Based Object Detection and Pose Estimation Revisited , 2015, 2015 International Conference on 3D Vision.

[34]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[35]  J. D. Wegner,et al.  Globally consistent registration of terrestrial laser scans via graph optimization , 2015 .

[36]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  Andrea Vedaldi,et al.  Instance Normalization: The Missing Ingredient for Fast Stylization , 2016, ArXiv.

[38]  Andrea Fusiello,et al.  Spectral Synchronization of Multiple Views in SE(3) , 2016, SIAM J. Imaging Sci..

[39]  Vladlen Koltun,et al.  Fast Global Registration , 2016, ECCV.

[40]  Jiaolong Yang,et al.  Go-ICP: A Globally Optimal Solution to 3D ICP Point-Set Registration , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[41]  Vincent Lepetit,et al.  Learning to Assign Orientations to Feature Points , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[42]  Vincent Lepetit,et al.  Going Further with Point Pair Features , 2016, ECCV.

[43]  Matthias Nießner,et al.  3DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructions , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[44]  Qi-Xing Huang,et al.  Translation Synchronization via Truncated Least Squares , 2017, NIPS.

[45]  Slobodan Ilic,et al.  CAD Priors for Accurate and Flexible Instance Reconstruction , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[46]  Vladlen Koltun,et al.  Learning Compact Geometric Features , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[47]  Luca Antiga,et al.  Automatic differentiation in PyTorch , 2017 .

[48]  Andrea Fusiello,et al.  Practical and Efficient Multi-view Matching , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[49]  Leonidas J. Guibas,et al.  PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[50]  Matthias Nießner,et al.  ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[51]  Slobodan Ilic,et al.  PPFNet: Global Context Aware Local Features for Robust 3D Point Matching , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[52]  Simon Korman,et al.  Latent RANSAC , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[53]  Vladlen Koltun,et al.  Deep Fundamental Matrix Estimation , 2018, ECCV.

[54]  Zi Jian Yew,et al.  3DFeat-Net: Weakly Supervised Local 3D Features for Point Cloud Registration , 2018, ECCV.

[55]  Vincent Lepetit,et al.  Learning to Find Good Correspondences , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[56]  Slobodan Ilic,et al.  PPF-FoldNet: Unsupervised Learning of Rotation Invariant 3D Local Descriptors , 2018, ECCV.

[57]  Venu Madhav Govindu,et al.  Robust Relative Rotation Averaging , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[58]  Stefan Roth,et al.  Neural Nearest Neighbors Networks , 2018, NeurIPS.

[59]  Slobodan Ilic,et al.  Bayesian Pose Graph Optimization via Bingham Distributions and Tempered Geodesic MCMC , 2018, NeurIPS.

[60]  Lei Zhou,et al.  Very Large-Scale Global SfM by Distributed Motion Averaging , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[61]  Umut Simsekli,et al.  Probabilistic Permutation Synchronization Using the Riemannian Structure of the Birkhoff Polytope , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[62]  Slobodan Ilic,et al.  3D Local Features for Direct Pairwise Registration , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[63]  Silvio Savarese,et al.  4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[64]  Shiyu Song,et al.  DeepVCP: An End-to-End Deep Neural Network for Point Cloud Registration , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[65]  Andreas Wieser,et al.  The Perfect Match: 3D Point Cloud Matching With Smoothed Densities , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[66]  Bastian Leibe,et al.  AlignNet-3D: Fast Point Cloud Registration of Partially Observed Objects , 2019, 2019 International Conference on 3D Vision (3DV).

[67]  Leonidas J. Guibas,et al.  Learning Transformation Synchronization , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[68]  Vladlen Koltun,et al.  Fully Convolutional Geometric Features , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[69]  Uttaran Bhattacharya,et al.  Efficient and Robust Registration on the 3D Special Euclidean Group , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[70]  Long Quan,et al.  Learning Two-View Correspondences and Geometry Using Order-Aware Network , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[71]  Yue Wang,et al.  Deep Closest Point: Learning Representations for Point Cloud Registration , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[72]  Chen Feng,et al.  DeepMapping: Unsupervised Map Estimation From Multiple Point Clouds , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).