Geometry-Consistent Neural Shape Representation with Implicit Displacement Fields

We present implicit displacement fields, a novel representation for detailed 3D geometry. Inspired by a classic surface deformation technique, displacement mapping, our method represents a complex surface as a smooth base surface plus a displacement along the base's normal directions, resulting in a frequency-based shape decomposition, where the high frequency signal is constrained geometrically by the low frequency signal. Importantly, this disentanglement is unsupervised thanks to a tailored architectural design that has an innate frequency hierarchy by construction. We explore implicit displacement field surface reconstruction and detail transfer and demonstrate superior representational power, training stability and generalizability.

[1]  Charles T. Loop,et al.  Neural Geometric Level of Detail: Real-time Rendering with Implicit 3D Shapes , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  Hao Zhang,et al.  DECOR-GAN: 3D Shape Detailization by Conditional Refinement , 2020, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Xiaolong Wang,et al.  Learning Continuous Image Representation with Local Implicit Image Function , 2020, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Hao Zhang,et al.  D2IM-Net: Learning Detail Disentangled Implicit Fields from Single Images , 2020, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Jiajun Wu,et al.  pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis , 2020, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  Francesc Moreno-Noguer,et al.  D-NeRF: Neural Radiance Fields for Dynamic Scenes , 2020, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Mohamed Elhoseiny,et al.  Adversarial Generation of Continuous Images , 2020, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  M. Zollhöfer,et al.  PatchNets: Patch-Based Generalizable Deep Implicit 3D Shape Representations , 2020, ECCV.

[9]  Kyaw Zaw Lin,et al.  Neural Sparse Voxel Fields , 2020, NeurIPS.

[10]  Raja Giryes,et al.  Deep geometric texture synthesis , 2020, ACM Trans. Graph..

[11]  Jonathan T. Barron,et al.  Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains , 2020, NeurIPS.

[12]  Gordon Wetzstein,et al.  Implicit Neural Representations with Periodic Activation Functions , 2020, NeurIPS.

[13]  Noah Snavely,et al.  DualSDF: Semantic Shape Manipulation Using a Two-Level Representation , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Hanbyul Joo,et al.  PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Ruigang Yang,et al.  FaceScape: A Large-Scale High Quality 3D Face Dataset and Detailed Riggable 3D Face Prediction , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Richard A. Newcombe,et al.  Deep Local Shapes: Learning Local SDF Priors for Detailed 3D Reconstruction , 2020, ECCV.

[17]  Ronen Basri,et al.  Multiview Neural Surface Reconstruction by Disentangling Geometry and Appearance , 2020, NeurIPS.

[18]  Pratul P. Srinivasan,et al.  NeRF , 2020, ECCV.

[19]  Thomas Funkhouser,et al.  Local Implicit Grid Representations for 3D Scenes , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Marc Pollefeys,et al.  Convolutional Occupancy Networks , 2020, ECCV.

[21]  Ronen Basri,et al.  Frequency Bias in Neural Networks for Input of Non-Uniform Density , 2020, ICML.

[22]  Andreas Geiger,et al.  Differentiable Volumetric Rendering: Learning Implicit 3D Representations Without 3D Supervision , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Vladimir G. Kim,et al.  Neural Cages for Detail-Preserving 3D Deformations , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Thomas Funkhouser,et al.  Local Deep Implicit Functions for 3D Shape , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Geoffrey E. Hinton,et al.  NASA: Neural Articulated Shape Approximation , 2019, ECCV.

[26]  Hao Zhang,et al.  BSP-Net: Generating Compact Meshes via Binary Space Partitioning , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  Geoffrey E. Hinton,et al.  CvxNet: Learnable Convex Decomposition , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Gordon Wetzstein,et al.  Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations , 2019, NeurIPS.

[29]  Hao Li,et al.  PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[30]  Thomas A. Funkhouser,et al.  Learning Shape Templates With Structured Implicit Functions , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[31]  Richard A. Newcombe,et al.  DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  Sebastian Nowozin,et al.  Occupancy Networks: Learning 3D Reconstruction in Function Space , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[33]  Hao Zhang,et al.  Learning Implicit Fields for Generative Shape Modeling , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  Zhi-Qin John Xu,et al.  Understanding training and generalization in deep learning by Fourier analysis , 2018, ArXiv.

[35]  Yoshua Bengio,et al.  Feature-wise transformations , 2018, Distill.

[36]  Zhi-Qin John Xu,et al.  Training behavior of deep neural network in frequency domain , 2018, ICONIP.

[37]  Yoshua Bengio,et al.  On the Spectral Bias of Neural Networks , 2018, ICML.

[38]  Aaron C. Courville,et al.  FiLM: Visual Reasoning with a General Conditioning Layer , 2017, AAAI.

[39]  Leonidas J. Guibas,et al.  PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space , 2017, NIPS.

[40]  Szymon Rusinkiewicz,et al.  Learning Detail Transfer based on Geometric Features , 2017, Comput. Graph. Forum.

[41]  Leonidas J. Guibas,et al.  PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[42]  Frank Hutter,et al.  SGDR: Stochastic Gradient Descent with Warm Restarts , 2016, ICLR.

[43]  Alec Jacobson,et al.  Thingi10K: A Dataset of 10, 000 3D-Printing Models , 2016, ArXiv.

[44]  Tamy Boubekeur,et al.  GeoBrush: Interactive Mesh Geometry Cloning , 2011, Comput. Graph. Forum.

[45]  Marco Attene,et al.  A lightweight approach to repairing digitized polygon meshes , 2010, The Visual Computer.

[46]  Pierre Alliez,et al.  Polygon Mesh Processing , 2010 .

[47]  James M. Rehg,et al.  Terrain Synthesis from Digital Elevation Models , 2007, IEEE Transactions on Visualization and Computer Graphics.

[48]  Kun Zhou,et al.  Mesh quilting for geometric texture synthesis , 2006, ACM Trans. Graph..

[49]  Christian Rössl,et al.  Laplacian surface editing , 2004, SGP '04.

[50]  H. Seidel,et al.  Multi-level partition of unity implicits , 2003, SIGGRAPH Courses.

[51]  Ioana M. Boier-Martin,et al.  Cut-and-paste editing of multiresolution surfaces , 2002, SIGGRAPH.

[52]  Henning Biermann,et al.  Texture and Shape Synthesis on Surfaces , 2001, Rendering Techniques.

[53]  Sarah F. Frisken,et al.  Adaptively sampled distance fields: a general representation of shape for computer graphics , 2000, SIGGRAPH.

[54]  Robert L. Cook,et al.  The Reyes image rendering architecture , 1987, SIGGRAPH.

[55]  S. Krishnamurthi,et al.  Shade Trees , 1899, The American Naturalist.

[56]  Daniel Cohen-Or,et al.  Progressive Encoding for Neural Optimization , 2021, ArXiv.

[57]  Dan B. Goldman,et al.  Non-parametric Texture Transfer Using MeshMatch , 2012 .