HyperPocket: Generative Point Cloud Completion

Scanning real-life scenes with modern registration devices typically give incomplete point cloud representations, mostly due to the limitations of the scanning process and 3D occlusions. Therefore, completing such partial representations remains a fundamental challenge of many computer vision applications. Most of the existing approaches aim to solve this problem by learning to reconstruct individual 3D objects in a synthetic setup of an uncluttered environment, which is far from a real-life scenario. In this work, we reformulate the problem of point cloud completion into an object hallucination task. Thus, we introduce a novel autoencoder-based architecture called HyperPocket that disentangles latent representations and, as a result, enables the generation of multiple variants of the completed 3D point clouds. We split point cloud processing into two disjoint data streams and leverage a hypernetwork paradigm to fill the spaces, dubbed pockets, that are left by the missing object parts. As a result, the generated point clouds are not only smooth but also plausible and geometrically consistent with the scene. Our method offers competitive performances to the other state-of-the-art models, and it enables a~plethora of novel applications.

[1]  Saifullahi Aminu Bello,et al.  Review: deep learning on 3D point clouds , 2020, Remote. Sens..

[2]  Jacek Tabor,et al.  HyperFlow: Representing 3D Objects as Surfaces , 2020, ArXiv.

[3]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[4]  Mathieu Aubry,et al.  A Papier-Mache Approach to Learning 3D Surface Generation , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[5]  Martial Hebert,et al.  PCN: Point Completion Network , 2018, 2018 International Conference on 3D Vision (3DV).

[6]  Matthias Nießner,et al.  Shape Completion Using 3D-Encoder-Predictor CNNs and Shape Synthesis , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Leonidas J. Guibas,et al.  ShapeNet: An Information-Rich 3D Model Repository , 2015, ArXiv.

[8]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[9]  Naveed Akhtar,et al.  Octree Guided CNN With Spherical Kernels for 3D Point Clouds , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[11]  Leonidas J. Guibas,et al.  PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space , 2017, NIPS.

[12]  Pieter Abbeel,et al.  Image Object Label 3 D CAD Model Candidate Grasps Google Object Recognition Engine Google Cloud Storage Select Feasible Grasp with Highest Success Probability Pose EstimationCamera Robots Cloud 3 D Sensor , 2014 .

[13]  Bence Nanay,et al.  The Importance of Amodal Completion in Everyday Perception , 2018, i-Perception.

[14]  Kaveh Hassani,et al.  Unsupervised Multi-Task Feature Learning on Point Clouds , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[15]  C. Qi.  Deep Learning on Point Sets for 3 D Classification and Segmentation , 2016 .

[16]  Dong Tian,et al.  FoldingNet: Point Cloud Auto-Encoder via Deep Grid Deformation , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[17]  Wei Wu,et al.  PointCNN: Convolution On X-Transformed Points , 2018, NeurIPS.

[18]  Kashif Rasul,et al.  Stochastic Maximum Likelihood Optimization via Hypernetworks , 2017, ArXiv.

[19]  Leonidas J. Guibas,et al.  PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Xiaogang Wang,et al.  Interpolated Convolutional Networks for 3D Point Cloud Understanding , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[21]  George Loizou,et al.  Computer vision and pattern recognition , 2007, Int. J. Comput. Math..

[22]  Ming Hao,et al.  Linked Dynamic Graph CNN: Learning on Point Cloud via Linking Hierarchical Features , 2019, ArXiv.

[23]  Lu Sheng,et al.  Morphing and Sampling Network for Dense Point Cloud Completion , 2019, AAAI.

[24]  Leonidas J. Guibas,et al.  Learning Representations and Generative Models for 3D Point Clouds , 2017, ICML.

[25]  Shangchen Zhou,et al.  GRNet: Gridding Residual Network for Dense Point Cloud Completion , 2020, ECCV.

[26]  Zhen Li,et al.  High-Resolution Shape Completion Using Deep Neural Networks for Global Structure and Local Geometry Inference , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[27]  Bin Yang,et al.  PIXOR: Real-time 3D Object Detection from Point Clouds , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[28]  Leonidas J. Guibas,et al.  PartNet: A Large-Scale Benchmark for Fine-Grained and Hierarchical Part-Level 3D Object Understanding , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  Niloy J. Mitra,et al.  Unpaired Point Cloud Completion on Real Scans using Adversarial Training , 2019, ICLR.

[30]  Rundi Wu,et al.  Multimodal Shape Completion via Conditional Generative Adversarial Networks , 2020, ECCV.

[31]  R. Venkatesh Babu,et al.  Dense 3D Point Cloud Reconstruction Using a Deep Pyramid Network , 2019, 2019 IEEE Winter Conference on Applications of Computer Vision (WACV).

[32]  Jacek Tabor,et al.  Hypernetwork Functional Image Representation , 2019, ICANN.

[33]  T. Trzciński,et al.  Hypernetwork approach to generating point clouds , 2020, ICML.

[34]  Kui Jia,et al.  Deep Cascade Generation on Point Sets , 2019, IJCAI.

[35]  Piotr Klukowski,et al.  Adversarial autoencoders for compact representations of 3D point clouds , 2018, Comput. Vis. Image Underst..

[36]  R. Briscoe.  MENTAL IMAGERY AND THE VARIETIES OF AMODAL PERCEPTION , 2011 .

[37]  Ming-Yu Liu,et al.  PointFlow: 3D Point Cloud Generation With Continuous Normalizing Flows , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).