SELF-SUPERVISED ADVERSARIAL SHAPE COMPLETION

Abstract. The goal of this paper is 3D shape completion: given an incomplete instance of a known category, hallucinate a complete version of it that is geometrically plausible. We develop an adversarial framework that makes it possible to learn shape completion in a self-supervised fashion, only from incomplete examples. This is enabled by a discriminator network that rejects incomplete shapes, via a loss function that separately assesses local sub-regions of the generated example and accepts only regions with sufficiently high point count. This inductive bias against empty regions forces the generator to output complete shapes. We demonstrate the effectiveness of this approach on synthetic data from ShapeNet and ModelNet, and on a real mobile mapping dataset with nearly 9’000 incomplete cars. Moreover, we apply it to the KITTI autonomous driving dataset without retraining, to highlight its ability to generalise to different data characteristics.

[1]  Bo Dai,et al.  Unsupervised 3D Shape Completion through GAN Inversion , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  Rundi Wu,et al.  Multimodal Shape Completion via Conditional Generative Adversarial Networks , 2020, ECCV.

[3]  Chenyang Lu,et al.  Hallucinating Beyond Observation: Learning to Complete with Partial Observation and Unpaired Prior Knowledge , 2019, ArXiv.

[4]  Niloy J. Mitra,et al.  Unpaired Point Cloud Completion on Real Scans using Adversarial Training , 2019, ICLR.

[5]  Di Sun,et al.  Symmetry-Aware Face Completion with Generative Adversarial Networks , 2018, ACCV.

[6]  Martial Hebert,et al.  PCN: Point Completion Network , 2018, 2018 International Conference on 3D Vision (3DV).

[7]  Andreas Geiger,et al.  Learning 3D Shape Completion from Laser Scan Data with Weak Supervision , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[8]  Yuichi Yoshida,et al.  Spectral Normalization for Generative Adversarial Networks , 2018, ICLR.

[9]  George Papandreou,et al.  Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation , 2018, ECCV.

[10]  Dong Tian,et al.  FoldingNet: Point Cloud Auto-Encoder via Deep Grid Deformation , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[11]  Jan Kautz,et al.  High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[12]  Zhen Li,et al.  High-Resolution Shape Completion Using Deep Neural Networks for Global Structure and Local Geometry Inference , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[13]  Sepp Hochreiter,et al.  GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium , 2017, NIPS.

[14]  Leonidas J. Guibas,et al.  PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space , 2017, NIPS.

[15]  Horst Bischof,et al.  OctNetFusion: Learning Depth Fusion from Data , 2017, 2017 International Conference on 3D Vision (3DV).

[16]  Sanja Fidler,et al.  Towards Diverse and Natural Image Descriptions via a Conditional GAN , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[17]  Leonidas J. Guibas,et al.  PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Matthias Nießner,et al.  Shape Completion Using 3D-Encoder-Predictor CNNs and Shape Synthesis , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Hao Li,et al.  High-Resolution Image Inpainting Using Multi-scale Neural Patch Synthesis , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Jiajun Wu,et al.  Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling , 2016, NIPS.

[22]  Max Jaderberg,et al.  Unsupervised Learning of 3D Structure from Images , 2016, NIPS.

[23]  Simon J. Julier,et al.  Structured Prediction of Unobserved Voxels from a Single Depth Image , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Oliver Grau,et al.  VConv-DAE: Deep Volumetric Shape Learning Without Object Labels , 2016, ECCV Workshops.

[25]  Sebastian Ramos,et al.  The Cityscapes Dataset for Semantic Urban Scene Understanding , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  Leonidas J. Guibas,et al.  ShapeNet: An Information-Rich 3D Model Repository , 2015, ArXiv.

[27]  Leonidas J. Guibas,et al.  Data-driven structural priors for shape completion , 2015, ACM Trans. Graph..

[28]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[29]  Simon Osindero,et al.  Conditional Generative Adversarial Nets , 2014, ArXiv.

[30]  Tobias Schreck,et al.  Approximate Symmetry Detection in Partial 3D Meshes , 2014, Comput. Graph. Forum.

[31]  Jianxiong Xiao,et al.  3D ShapeNets: A deep representation for volumetric shapes , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  Aaron C. Courville,et al.  Generative Adversarial Nets , 2014, NIPS.

[33]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[34]  Niloy J. Mitra,et al.  Symmetry in 3D Geometry: Extraction and Applications , 2013, Comput. Graph. Forum.

[35]  Pascal Getreuer,et al.  Total Variation Inpainting using Split Bregman , 2012, Image Process. Line.

[36]  Leonidas J. Guibas,et al.  Discovering structural regularity in 3D geometry , 2008, ACM Trans. Graph..

[37]  Leonidas J. Guibas,et al.  Partial and approximate symmetry detection for 3D geometry , 2006, ACM Trans. Graph..

[38]  T. Funkhouser,et al.  A planar-reflective symmetry transform for 3D shapes , 2006, ACM Trans. Graph..

[39]  Alla Sheffer,et al.  Template-based mesh completion , 2005, SGP '05.

[40]  S. Osher,et al.  Fast surface reconstruction using the level set method , 2001, Proceedings IEEE Workshop on Variational and Level Set Methods in Computer Vision.

[41]  A Ankitha,et al.  Exemplar Based Image Inpainting , 2014 .

[42]  Leonidas J. Guibas,et al.  Example-Based 3D Scan Completion , 2005 .