A Critical Analysis of NeRF-Based 3D Reconstruction

This paper presents a critical analysis of image-based 3D reconstruction using neural radiance fields (NeRFs), with a focus on quantitative comparisons with respect to traditional photogrammetry. The aim is, therefore, to objectively evaluate the strengths and weaknesses of NeRFs and provide insights into their applicability to different real-life scenarios, from small objects to heritage and industrial scenes. After a comprehensive overview of photogrammetry and NeRF methods, highlighting their respective advantages and disadvantages, various NeRF methods are compared using diverse objects with varying sizes and surface characteristics, including texture-less, metallic, translucent, and transparent surfaces. We evaluated the quality of the resulting 3D reconstructions using multiple criteria, such as noise level, geometric accuracy, and the number of required images (i.e., image baselines). The results show that NeRFs exhibit superior performance over photogrammetry in terms of non-collaborative objects with texture-less, reflective, and refractive surfaces. Conversely, photogrammetry outperforms NeRFs in cases where the object’s surface possesses cooperative texture. Such complementarity should be further exploited in future works.

[1]  A. Karami,et al.  NERF FOR HERITAGE 3D RECONSTRUCTION , 2023, The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences.

[2]  P. Véron,et al.  NEURAL RADIANCE FIELDS (NERF): REVIEW AND POTENTIAL APPLICATIONS TO DIGITAL CULTURAL HERITAGE , 2023, The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences.

[3]  M. Varshosaz,et al.  FFT-BASED FILTERING APPROACH TO FUSE PHOTOGRAMMETRY AND PHOTOMETRIC STEREO 3D DATA , 2023, ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences.

[4]  F. Menna,et al.  PHOTOGRAMMETRY NOW AND THEN – FROM HAND-CRAFTED TO DEEP-LEARNING TIE POINTS – , 2022, The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences.

[5]  Fabio Remondino,et al.  Combining Photogrammetry and Photometric Stereo to Achieve Precise and Complete 3D Reconstruction , 2022, Sensors.

[6]  S. Fidler,et al.  Variable Bitrate Neural Fields , 2022, SIGGRAPH.

[7]  Andreas Geiger,et al.  MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction , 2022, NeurIPS.

[8]  F. Menna,et al.  3D DIGITIZATION OF TRANSPARENT AND GLASS SURFACES: STATE OF THE ART AND ANALYSIS OF SOME METHODS , 2022, The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences.

[9]  M. Varshosaz,et al.  Exploiting Light Directionality for Image‐Based 3D Reconstruction of Non‐Collaborative Surfaces , 2022, The Photogrammetric Record.

[10]  N. Pfeifer,et al.  PROJECT INDIGO – DOCUMENT, DISSEMINATE & ANALYSE A GRAFFITI-SCAPE , 2022, The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences.

[11]  T. Müller,et al.  Instant neural graphics primitives with a multiresolution hash encoding , 2022, ACM Trans. Graph..

[12]  Erion Plaku,et al.  Joint computational design of workspaces and workplans , 2021, ACM Trans. Graph..

[13]  Bing Liu,et al.  Multi-view stereo in the Deep Learning Era: A comprehensive revfiew , 2021, Displays.

[14]  J.-Y. Zhu,et al.  Advances in Neural Rendering , 2021, SIGGRAPH Courses.

[15]  Jaehyun Lee,et al.  Evaluating Feature Extraction Methods with Synthetic Noise Patterns for Image-Based Modelling of Texture-Less Objects , 2020, Remote. Sens..

[16]  Erica Nocerino,et al.  Surface Reconstruction Assessment in Photogrammetric Applications , 2020, Sensors.

[17]  Kin-Man Lam,et al.  Pay Attention to Devils: A Photometric Stereo Network for Better Details , 2020, IJCAI.

[18]  Jonathan T. Barron,et al.  Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains , 2020, NeurIPS.

[19]  Gordon Wetzstein,et al.  Implicit Neural Representations with Periodic Activation Functions , 2020, NeurIPS.

[20]  Gordon Wetzstein,et al.  State of the Art on Neural Rendering , 2020, Comput. Graph. Forum.

[21]  Pratul P. Srinivasan,et al.  NeRF , 2020, ECCV.

[22]  Boxin Shi,et al.  Multi-View Photometric Stereo: A Robust Solution and Benchmark Dataset for Spatially Varying Isotropic Materials , 2020, IEEE Transactions on Image Processing.

[23]  Mario Sokac,et al.  Evaluation of synthetically generated patterns for image-based 3D reconstruction of texture-less objects , 2019 .

[24]  Ali Karami,et al.  An automatic 3D reconstruction system for texture-less objects , 2019, Robotics Auton. Syst..

[25]  Minglun Gong,et al.  Full 3D reconstruction of transparent objects , 2018, ACM Trans. Graph..

[26]  Svorad Stolc,et al.  A Review of Depth and Normal Fusion Algorithms , 2018, Sensors.

[27]  Fabio Menna,et al.  An Open Source Low-Cost Automatic System for Image-Based 3d Digitization , 2017 .

[28]  Yasuyuki Matsushita,et al.  Robust Multiview Photometric Stereo Using Planar Mesh Parameterization , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[29]  ARNO KNAPITSCH,et al.  Tanks and temples , 2017, ACM Trans. Graph..

[30]  Marco Callieri,et al.  3d Digitization of an Heritage Masterpiece - A Critical Analysis on Quality Assessment , 2016 .

[31]  Fabrizio Ivan Apollonio,et al.  An Advanced Pre-Processing Pipeline to Improve Automated Photogrammetric Reconstructions of Architectural Scenes , 2016, Remote. Sens..

[32]  Masood Varshosaz,et al.  THE PERFORMANCE EVALUATION OF MULTI-IMAGE 3D RECONSTRUCTION SOFTWARE WITH DIFFERENT SENSORS , 2015 .

[33]  Diego González-Aguilera,et al.  Procedure for quality inspection of welds based on macro-photogrammetric three-dimensional reconstruction , 2015 .

[34]  Fabio Menna,et al.  Photogrammetry applied to Problematic artefacts , 2014 .

[35]  Jan Boehm,et al.  Close-Range Photogrammetry and 3D Imaging , 2013 .

[36]  Fabio Remondino,et al.  Heritage Recording and 3D Modeling with Photogrammetry and 3D Scanning , 2011, Remote. Sens..

[37]  Giovanna Sansoni,et al.  State-of-The-Art and Applications of 3D Imaging Sensors in Industry, Cultural Heritage, Medicine, and Criminal Investigation , 2009, Sensors.

[38]  Samik Raychaudhuri,et al.  Introduction to Monte Carlo simulation , 2008, 2008 Winter Simulation Conference.

[39]  Fabio Remondino,et al.  Image‐based 3D Modelling: A Review , 2006 .

[40]  C. S. Fraser,et al.  INDUSTRIAL PHOTOGRAMMETRY: NEW DEVELOPMENTS AND RECENT APPLICATIONS , 2006 .

[41]  Richard Szeliski,et al.  A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[42]  Larry S. Davis,et al.  A probabilistic framework for surface reconstruction from multiple images , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[43]  Paul J. Besl,et al.  A Method for Registration of 3-D Shapes , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[44]  William E. Lorensen,et al.  Marching cubes: A high resolution 3D surface construction algorithm , 1987, SIGGRAPH.

[45]  Robert J. Woodham,et al.  Photometric method for determining surface orientation from multiple images , 1980 .

[46]  A. Karami,et al.  INVESTIGATING 3D RECONSTRUCTION OF NON-COLLABORATIVE SURFACES THROUGH PHOTOGRAMMETRY AND PHOTOMETRIC STEREO , 2021 .

[47]  Bijan Samali,et al.  Quality Evaluation of Digital Twins Generated Based on UAV Photogrammetry and TLS: Bridge Case Study , 2021, Remote. Sens..