BeyondPixels: A Comprehensive Review of the Evolution of Neural Radiance Fields

Neural rendering combines ideas from classical computer graphics and machine learning to synthesize images from real-world observations. NeRF, short for Neural Radiance Fields, is a recent innovation that uses AI algorithms to create 3D objects from 2D images. By leveraging an interpolation approach, NeRF can produce new 3D reconstructed views of complicated scenes. Rather than directly restoring the whole 3D scene geometry, NeRF generates a volumetric representation called a ``radiance field,'' which is capable of creating color and density for every point within the relevant 3D space. The broad appeal and notoriety of NeRF make it imperative to examine the existing research on the topic comprehensively. While previous surveys on 3D rendering have primarily focused on traditional computer vision-based or deep learning-based approaches, only a handful of them discuss the potential of NeRF. However, such surveys have predominantly focused on NeRF's early contributions and have not explored its full potential. NeRF is a relatively new technique continuously being investigated for its capabilities and limitations. This survey reviews recent advances in NeRF and categorizes them according to their architectural designs, especially in the field of novel view synthesis.

[1]  P. Spurek,et al.  HyperNeRFGAN: Hypernetwork approach to 3D NeRF GAN , 2023, ArXiv.

[2]  Yue Wang,et al.  SteerNeRF: Accelerating NeRF Rendering via Smooth Viewpoint Trajectory , 2022, 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Sungheon Park,et al.  PANeRF: Pseudo-view Augmentation for Improved Neural Radiance Fields Based on Few-shot Inputs , 2022, ArXiv.

[4]  Yi-Ling Qiao,et al.  NeuPhysics: Editable Neural Geometry and Physics from Monocular Videos , 2022, NeurIPS.

[5]  Gang Yu,et al.  Coordinates Are NOT Lonely - Codebook Prior Helps Implicit Neural 3D Representations , 2022, NeurIPS.

[6]  Linlin Xu,et al.  NeRF: Neural Radiance Field in 3D Vision, A Comprehensive Review , 2022, 2210.00379.

[7]  Muhammad Zeshan Afzal,et al.  Three-Dimensional Reconstruction from a Single RGB Image Using Deep Learning: A Review , 2022, J. Imaging.

[8]  Can Wang,et al.  FDNeRF: Few-shot Dynamic Neural Radiance Fields for Face Reconstruction and Expression Editing , 2022, SIGGRAPH Asia.

[9]  T. Funkhouser,et al.  MobileNeRF: Exploiting the Polygon Rasterization Pipeline for Efficient Neural Field Rendering on Mobile Architectures , 2022, 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Tatsuya Harada,et al.  Deforming Radiance Fields with Cages , 2022, ECCV.

[11]  Connor Z. Lin,et al.  NeuForm: Adaptive Overfitting for Neural Shape Editing , 2022, NeurIPS.

[12]  Andreas Geiger,et al.  VoxGRAF: Fast 3D-Aware Image Synthesis with Sparse Voxel Grids , 2022, NeurIPS.

[13]  Chuang Gan,et al.  SNAKE: Shape-aware Neural 3D Keypoint Field , 2022, NeurIPS.

[14]  Peter R. Florence,et al.  Reinforcement Learning with Neural Radiance Fields , 2022, NeurIPS.

[15]  Jiaya Jia,et al.  EfficientNeRF - Efficient Neural Radiance Fields , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Mike Zheng Shou,et al.  DeVRF: Fast Deformable Voxel Radiance Fields for Dynamic Scenes , 2022, NeurIPS.

[17]  A. Tagliasacchi,et al.  D$^2$NeRF: Self-Supervised Decoupling of Dynamic and Static Objects from a Monocular Video , 2022, 2205.15838.

[18]  V. Sitzmann,et al.  Decomposing NeRF for Editing via Feature Field Distillation , 2022, NeurIPS.

[19]  Yu-Kun Lai,et al.  NeRF-Editing: Geometry Editing of Neural Radiance Fields , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  M. Nießner,et al.  AutoRF: Learning 3D Object Radiance Fields from Single View Observations , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  M. Tarr,et al.  Learning Neural Acoustic Fields , 2022, NeurIPS.

[22]  Hao Su,et al.  NeRFusion: Fusing Radiance Fields for Large-Scale Scene Reconstruction , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Slobodan Ilic,et al.  NeRF-Pose: A First-Reconstruct-Then-Regress Approach for Weakly-supervised 6D Object Pose Estimation , 2022, 2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW).

[24]  L. Gool,et al.  Pix2NeRF: Unsupervised Conditional $\pi$-GAN for Single Image to Neural Radiance Fields Translation , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Pratul P. Srinivasan,et al.  Block-NeRF: Scalable Large Scene Neural View Synthesis , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  Hubert P. H. Shum,et al.  MedNeRF: Medical Neural Radiance Fields for Reconstructing 3D-aware CT-Projections from a Single X-ray , 2022, 2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC).

[27]  U. Neumann,et al.  Point-NeRF: Point-based Neural Radiance Fields , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  T. Müller,et al.  Instant neural graphics primitives with a multiresolution hash encoding , 2022, ACM Trans. Graph..

[29]  Pratul P. Srinivasan,et al.  HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Video , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  A. Vedaldi,et al.  BANMo: Building Animatable 3D Neural Models from Many Casual Videos , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[31]  D. Ramanan,et al.  Mega-NeRF: Scalable Construction of Large-Scale NeRFs for Virtual Fly- Throughs , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  Shalini De Mello,et al.  Efficient Geometry-aware 3D Generative Adversarial Networks , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[33]  Benjamin Recht,et al.  Plenoxels: Radiance Fields without Neural Networks , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  Jonathan T. Barron,et al.  Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fields , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[35]  Jiakai Zhang,et al.  HumanNeRF: Efficiently Generated Human Radiance Field from Sparse Inputs , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[36]  Marek Kowalski,et al.  CoNeRF: Controllable Neural Radiance Fields , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  P. Abbeel,et al.  Zero-Shot Text-Guided Object Generation with Dream Fields , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[38]  Yebin Liu,et al.  FENeRF: Face Editing in Neural Radiance Fields , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[39]  Jonathan T. Barron,et al.  NeRF in the Dark: High Dynamic Range View Synthesis from Noisy Raw Images , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[40]  Hwann-Tzong Chen,et al.  Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[41]  K. M. Yi,et al.  LOLNeRF: Learn from One Look , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[42]  Daniel Keysers,et al.  LiT: Zero-Shot Transfer with Locked-image text Tuning , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[43]  Sida Peng,et al.  Towards efficient and photorealistic 3D human reconstruction: A brief survey , 2021, Visual Informatics.

[44]  Hujun Bao,et al.  Learning Object-Compositional Neural Radiance Field for Editable Scene Rendering , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[45]  J.-Y. Zhu,et al.  Advances in Neural Rendering , 2021, SIGGRAPH Courses.

[46]  Hui Huang,et al.  Capturing, Reconstructing, and Simulating: The UrbanScene3D Dataset , 2021, ECCV.

[47]  Huchuan Lu,et al.  Animatable Neural Radiance Fields from Monocular RGB Video , 2021, ArXiv.

[48]  Zhoutong Zhang,et al.  Editing Conditional Radiance Fields , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[49]  Ruth Wario,et al.  A survey on deep learning in medical image reconstruction , 2021 .

[50]  Antonio Torralba,et al.  BARF: Bundle-Adjusting Neural Radiance Fields , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[51]  Stephen Lin,et al.  Neural Articulated Radiance Field , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[52]  Jingyi Yu,et al.  MirrorNeRF: One-shot Neural Portrait Radiance Field from Multi-mirror Catadioptric Imaging , 2021, 2021 IEEE International Conference on Computational Photography (ICCP).

[53]  Adam R. Kosiorek,et al.  NeRF-VAE: A Geometry Aware 3D Scene Generative Model , 2021, ICML.

[54]  Pieter Abbeel,et al.  Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[55]  Andreas Geiger,et al.  CAMPARI: Camera-Aware Decomposed Generative Neural Radiance Fields , 2021, 2021 International Conference on 3D Vision (3DV).

[56]  Hao Su,et al.  GNeRF: GAN-based Neural Radiance Field without Posed Camera , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[57]  Jonathan T. Barron,et al.  Baking Neural Radiance Fields for Real-Time View Synthesis , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[58]  Yiyi Liao,et al.  KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[59]  Ren Ng,et al.  PlenOctrees for Real-time Rendering of Neural Radiance Fields , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[60]  Pratul P. Srinivasan,et al.  Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[61]  Stephan J. Garbin,et al.  FastNeRF: High-Fidelity Neural Rendering at 200FPS , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[62]  Ilya Sutskever,et al.  Learning Transferable Visual Models From Natural Language Supervision , 2021, ICML.

[63]  V. Prisacariu,et al.  NeRF-: Neural Radiance Fields Without Known Camera Parameters , 2021, ArXiv.

[64]  Tanner Schmidt,et al.  STaR: Self-supervised Tracking and Reconstruction of Rigid Objects in Motion with Neural Rendering , 2020, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[65]  Frank Dellaert,et al.  Neural Volume Rendering: NeRF And Beyond , 2020, ArXiv.

[66]  Jonathan T. Barron,et al.  iNeRF: Inverting Neural Radiance Fields for Pose Estimation , 2020, 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[67]  Jonathan T. Barron,et al.  NeRD: Neural Reflectance Decomposition from Image Collections , 2020, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[68]  Angjoo Kanazawa,et al.  pixelNeRF: Neural Radiance Fields from One or Few Images , 2020, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[69]  Jiajun Wu,et al.  pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis , 2020, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[70]  Andreas Geiger,et al.  GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields , 2020, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[71]  Andrea Vedaldi,et al.  Continuous Surface Embeddings , 2020, NeurIPS.

[72]  Kai Zhang,et al.  NeRF++: Analyzing and Improving Neural Radiance Fields , 2020, ArXiv.

[73]  Alex Trevithick,et al.  GRF: Learning a General Radiance Field for 3D Representation and Rendering , 2020, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[74]  Susmita Sur-Kolay,et al.  3D reconstruction of spine image from 2D MRI slices along one axis , 2020, IET Image Process..

[75]  Jonathan T. Barron,et al.  NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections , 2020, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[76]  Daniel Cohen-Or,et al.  Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation , 2020, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[77]  Tetsuya Matsuda,et al.  X-ray2Shape: Reconstruction of 3D Liver Shape from a Single 2D Projection Image , 2020, 2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC).

[78]  Adrien Gaidon,et al.  Differentiable Rendering: A Survey , 2020, ArXiv.

[79]  G. Hamarneh,et al.  Deep learning for biomedical image reconstruction: a survey , 2020, Artificial Intelligence Review.

[80]  Linlin Shen,et al.  3D Neuron Reconstruction in Tangled Neuronal Image With Deep Networks , 2020, IEEE Transactions on Medical Imaging.

[81]  Michael J. Black,et al.  VIBE: Video Inference for Human Body Pose and Shape Estimation , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[82]  Tero Karras,et al.  Analyzing and Improving the Image Quality of StyleGAN , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[83]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[84]  Hendra Hendra,et al.  Computer Vision Based 3D Reconstruction : A Review , 2019, International Journal of Electrical and Computer Engineering (IJECE).

[85]  Nanik Suciati,et al.  A Review of Deep Learning Techniques for 3D Reconstruction of 2D Images , 2019, 2019 12th International Conference on Information & Communication Technology and System (ICTS).

[86]  Gordon Wetzstein,et al.  Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations , 2019, NeurIPS.

[87]  Richard A. Newcombe,et al.  DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[88]  Sebastian Nowozin,et al.  Occupancy Networks: Learning 3D Reconstruction in Function Space , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[89]  Muhammad Abid,et al.  A Methodological Review of 3D Reconstruction Techniques in Tomographic Imaging , 2018, Journal of Medical Systems.

[90]  Jaakko Lehtinen,et al.  Progressive Growing of GANs for Improved Quality, Stability, and Variation , 2017, ICLR.

[91]  Song Tian,et al.  Comparison of medical image 3D reconstruction rendering methods for robot-assisted surgery , 2017, 2017 2nd International Conference on Advanced Robotics and Mechatronics (ICARM).

[92]  Joon Son Chung,et al.  VoxCeleb: A Large-Scale Speaker Identification Dataset , 2017, INTERSPEECH.

[93]  Jack J. Dongarra,et al.  Novel HPC techniques to batch execution of many variable size BLAS computations on GPUs , 2017, ICS.

[94]  Leonidas J. Guibas,et al.  PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[95]  Gernot Riegler,et al.  OctNet: Learning Deep 3D Representations at High Resolutions , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[96]  Riyanto Sigit,et al.  3D heart image reconstruction and visualization with marching cubes algorithm , 2016, 2016 International Conference on Knowledge Creation and Intelligent Computing (KCIC).

[97]  Michael J. Black,et al.  SMPL: A Skinned Multi-Person Linear Model , 2023 .

[98]  Carlos Hernandez,et al.  Multi-View Stereo: A Tutorial , 2015, Found. Trends Comput. Graph. Vis..

[99]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[100]  Henrik Aanæs,et al.  Large Scale Multi-view Stereopsis Evaluation , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[101]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.

[102]  Eko Supriyanto,et al.  Development of 3D image reconstruction based on untracked 2D fetal phantom ultrasound images using VTK , 2010 .

[103]  William E. Lorensen,et al.  Marching cubes: A high resolution 3D surface construction algorithm , 1987, SIGGRAPH.

[104]  E. Shlizerman,et al.  INRAS: Implicit Neural Representation for Audio Scenes , 2022, NeurIPS.

[105]  Y. Cheng,et al.  CageNeRF: Cage-based Neural Radiance Field for Generalized 3D Deformation and Animation , 2022, NeurIPS.

[106]  M. Aharchi,et al.  A Review on 3D Reconstruction Techniques from 2D Images , 2020 .

[107]  Ulus Çevik,et al.  Three-Dimensional Modeling in Medical Image Processing by Using Fractal Geometry , 2017, J. Comput..

[108]  Conference on Computer Vision and Pattern Recognition, CVPR 1993, 15-17 June, 1993, New York, NY, USA , 1993 .