Multi-scale hash encoding based neural geometry representation

Recently, neural implicit function-based representation has attracted more and more attention, and has been widely used to represent surfaces together with differentiable neural networks. However, existing neural geometry representations still suffer from slow computation speed and insufficient reconstruction accuracy when applied to surface reconstructions from point clouds and multi-view images. To alleviate these issues, we propose a multi-scale hash encoding-based neural geometry representation to effectively and efficiently optimize the surface represented as a signed distance field. To this end, a novel neural network structure is proposed by carefully combining low-frequency Fourier position encoding with multi-scale hash encoding. Accordingly, the initialization of the geometry network and geometry features of the rendering module is redesigned. Extensive experiments demonstrate that our proposed representation achieves at least 10 times speedup on the task of million-level point cloud reconstruction, and significantly improves the efficiency and accuracy on the multi-view reconstruction task. Our code and models will be available at https://github.com/Dengzhi-USTC/ Neural-Geometry-Reconstruction .

[1]  Juyong Zhang,et al.  Reconstructing Personalized Semantic Facial NeRF Models from Monocular Video , 2022, ACM Trans. Graph..

[2]  Yan Wang,et al.  Neural Surface Reconstruction of Dynamic Scenes with Monocular RGB-D Camera , 2022, NeurIPS.

[3]  R. Raskar,et al.  Towards Learning Neural Representations from Shadows , 2022, ECCV.

[4]  T. Funkhouser,et al.  Neural dual contouring , 2022, ACM Trans. Graph..

[5]  H. Bao,et al.  SelfRecon: Self Reconstruction Your Digital Avatar from Monocular Video , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  N. Vijaykumar,et al.  CoordX: Accelerating Implicit Neural Representation with a Split MLP Architecture , 2022, ICLR.

[7]  Bailin Deng,et al.  Sketch2PQ: Freeform Planar Quadrilateral Mesh Design via a Single Sketch , 2022, IEEE Transactions on Visualization and Computer Graphics.

[8]  T. Müller,et al.  Instant neural graphics primitives with a multiresolution hash encoding , 2022, ACM Trans. Graph..

[9]  Long Quan,et al.  Learning Signed Distance Field for Multi-view Surface Reconstruction , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[10]  Yudong Guo,et al.  Prior-Guided Multi-View 3D Head Reconstruction , 2021, IEEE Transactions on Multimedia.

[11]  Yaron Lipman,et al.  Volume Rendering of Neural Implicit Surfaces , 2021, NeurIPS.

[12]  C. Theobalt,et al.  NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction , 2021, NeurIPS.

[13]  Yang Liu,et al.  Spline Positional Encoding for Learning 3D Implicit Signed Distance Fields , 2021, IJCAI.

[14]  Gordon Wetzstein,et al.  Acorn , 2021, ACM Trans. Graph..

[15]  Qi Shan,et al.  MVS2D: Efficient Multiview Stereo via Attention-Driven 2D Convolutions , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  R. Giryes,et al.  SAPE: Spatially-Adaptive Progressive Encoding for Neural Optimization , 2021, NeurIPS.

[17]  Hujun Bao,et al.  NeuralRecon: Real-Time Coherent 3D Reconstruction from Monocular Video , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Yang Liu,et al.  Deep Implicit Moving Least-Squares Functions for 3D Reconstruction , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Charles T. Loop,et al.  Neural Geometric Level of Detail: Real-time Rendering with Implicit 3D Shapes , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Shimin Hu,et al.  DI-Fusion: Online Implicit 3D Reconstruction with Deep Priors , 2020, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Silvano Galliani,et al.  PatchmatchNet: Learned Multi-View Patchmatch Stereo , 2020, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Matthias Zwicker,et al.  Neural-Pull: Learning Signed Distance Functions from Point Clouds by Learning to Pull Space onto Surfaces , 2020, ICML.

[23]  Mehrtash Harandi,et al.  Hierarchical Neural Architecture Search for Deep Stereo Matching , 2020, NeurIPS.

[24]  Gerard Pons-Moll,et al.  Neural Unsigned Distance Fields for Implicit Function Learning , 2020, NeurIPS.

[25]  N. Mitra,et al.  Points2Surf Learning Implicit Surfaces from Point Clouds , 2020, ECCV.

[26]  Jonathan T. Barron,et al.  Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains , 2020, NeurIPS.

[27]  Gordon Wetzstein,et al.  Implicit Neural Representations with Periodic Activation Functions , 2020, NeurIPS.

[28]  Yaron Lipman,et al.  SAL++: Sign Agnostic Learning with Derivatives , 2020, ArXiv.

[29]  Juyong Zhang,et al.  AANet: Adaptive Aggregation Network for Efficient Stereo Matching , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Ruigang Yang,et al.  FaceScape: A Large-Scale High Quality 3D Face Dataset and Detailed Riggable 3D Face Prediction , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[31]  Richard A. Newcombe,et al.  Deep Local Shapes: Learning Local SDF Priors for Detailed 3D Reconstruction , 2020, ECCV.

[32]  Ronen Basri,et al.  Multiview Neural Surface Reconstruction by Disentangling Geometry and Appearance , 2020, NeurIPS.

[33]  Marc Pollefeys,et al.  Convolutional Occupancy Networks , 2020, ECCV.

[34]  Y. Lipman,et al.  Implicit Geometric Regularization for Learning Shapes , 2020, ICML.

[35]  Yu-Kun Lai,et al.  A survey on deep geometry learning: From a representation perspective , 2020, Computational Visual Media.

[36]  J. Álvarez,et al.  Cost Volume Pyramid Based Depth Inference for Multi-View Stereo , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  Andreas Geiger,et al.  Differentiable Volumetric Rendering: Learning Implicit 3D Representations Without 3D Supervision , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[38]  Thomas Funkhouser,et al.  Local Deep Implicit Functions for 3D Shape , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[39]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[40]  Y. Lipman,et al.  SAL: Sign Agnostic Learning of Shapes From Raw Data , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[41]  Long Quan,et al.  BlendedMVS: A Large-Scale Dataset for Generalized Multi-View Stereo Networks , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[42]  Jing Xu,et al.  Point-Based Multi-View Stereo Network , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[43]  Long Quan,et al.  Recurrent MVSNet for High-Resolution Multi-View Stereo Depth Inference , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[44]  Richard A. Newcombe,et al.  DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[45]  Sebastian Nowozin,et al.  Occupancy Networks: Learning 3D Reconstruction in Function Space , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[46]  Hao Zhang,et al.  Learning Implicit Fields for Generative Shape Modeling , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[47]  Yoshua Bengio,et al.  On the Spectral Bias of Neural Networks , 2018, ICML.

[48]  Jitendra Malik,et al.  Learning a Multi-View Stereo Machine , 2017, NIPS.

[49]  Pierre Alliez,et al.  A Survey of Surface Reconstruction from Point Clouds , 2017, Comput. Graph. Forum.

[50]  Jan-Michael Frahm,et al.  Pixelwise View Selection for Unstructured Multi-View Stereo , 2016, ECCV.

[51]  Michael Goesele,et al.  Shading-Aware Multi-view Stereo , 2016, ECCV.

[52]  Carlos Hernandez,et al.  Multi-View Stereo: A Tutorial , 2015, Found. Trends Comput. Graph. Vis..

[53]  Henrik Aanæs,et al.  Large Scale Multi-view Stereopsis Evaluation , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[54]  Matthias Nießner,et al.  Real-time 3D reconstruction at scale using voxel hashing , 2013, ACM Trans. Graph..

[55]  Eitan Grinspun,et al.  Discrete laplace operators: no free lunch , 2007, Symposium on Geometry Processing.

[56]  J. Ponce,et al.  Accurate, Dense, and Robust Multi-View Stereopsis , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[57]  Bernhard Schölkopf,et al.  Implicit Surfaces with Globally Regularised and Compactly Supported Basis Functions , 2006, NIPS.

[58]  Michael M. Kazhdan,et al.  Poisson surface reconstruction , 2006, SGP '06.

[59]  Richard K. Beatson,et al.  Reconstruction and representation of 3D objects with radial basis functions , 2001, SIGGRAPH.

[60]  William E. Lorensen,et al.  Marching cubes: A high resolution 3D surface construction algorithm , 1987, SIGGRAPH.

[61]  J. Martin,et al.  Reconstruction , 1980, The Lancet.