FusedRF: Fusing Multiple Radiance Fields

Radiance Fields (RFs) have shown great potential to represent scenes from casually captured discrete views. Compositing parts or whole of multiple captured scenes could greatly interest several XR applications. Prior works can generate new views of such scenes by tracing each scene in parallel. This increases the render times and memory requirements with the number of components. In this work, we provide a method to create a single, compact, fused RF representation for a scene composited using multiple RFs. The fused RF has the same render times and memory utilizations as a single RF. Our method distills information from multiple teacher RFs into a single student RF while also facilitating further manipulations like addition and deletion into the fused representation.

[1]  Ye Pan,et al.  Interactive Geometry Editing of Neural Radiance Fields , 2023, ArXiv.

[2]  Mikaela Angelina Uy,et al.  Generating Part-Aware Editable 3D Shapes without 3D Supervision , 2023, 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Rahul Goel,et al.  Interactive Segmentation of Radiance Fields , 2022, 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Bolei Zhou,et al.  DisCoScene: Spatially Disentangled Generative Radiance Fields for Controllable 3D-aware Scene Synthesis , 2022, 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  M. Pollefeys,et al.  Removing Objects From Neural Radiance Fields , 2022, 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  F. Xu,et al.  EditableNeRF: Editing Topologically Varying Neural Radiance Fields by Key Points , 2022, 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Shuchang Zhou,et al.  One is All: Bridging the Gap Between Neural Radiance Fields Architectures with Progressive Volume Distillation , 2022, AAAI.

[8]  Marcus A. Brubaker,et al.  SPIn-NeRF: Multiview Segmentation and Perceptual Inpainting with Neural Radiance Fields , 2022, 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  László A. Jeni,et al.  CoNFies: Controllable Neural Face Avatars , 2022, 2023 IEEE 17th International Conference on Automatic Face and Gesture Recognition (FG).

[10]  Chaojian Li,et al.  RT-NeRF: Real-Time On-Device Neural Radiance Fields Towards Immersive AR/VR Rendering , 2022, 2022 IEEE/ACM International Conference On Computer Aided Design (ICCAD).

[11]  Yi-Ling Qiao,et al.  NeuPhysics: Editable Neural Geometry and Physics from Monocular Videos , 2022, NeurIPS.

[12]  N. Vijaykumar,et al.  SPIDR: SDF-based Neural Point Fields for Illumination and Deformation , 2022, ArXiv.

[13]  A. Vedaldi,et al.  Neural Feature Fusion Fields: 3D Distillation of Self-Supervised 2D Image Representations , 2022, 2022 International Conference on 3D Vision (3DV).

[14]  A. Tagliasacchi,et al.  Volume Rendering Digest (for NeRF) , 2022, ArXiv.

[15]  Tatsuya Harada,et al.  Deforming Radiance Fields with Cages , 2022, ECCV.

[16]  H. Bao,et al.  NeuMesh: Learning Disentangled Neural Mesh-based Implicit Field for Geometry and Texture Editing , 2022, ECCV.

[17]  Jianmin Zheng,et al.  Object-Compositional Neural Implicit Surfaces , 2022, ECCV.

[18]  Thu Nguyen-Phuoc,et al.  SNeRF , 2022, ACM Trans. Graph..

[19]  Hwann-Tzong Chen,et al.  Improved Direct Voxel Grid Optimization for Radiance Fields Reconstruction , 2022, ArXiv.

[20]  A. Tagliasacchi,et al.  D$^2$NeRF: Self-Supervised Decoupling of Dynamic and Static Objects from a Monocular Video , 2022, 2205.15838.

[21]  V. Sitzmann,et al.  Decomposing NeRF for Editing via Feature Field Distillation , 2022, NeurIPS.

[22]  Yu-Kun Lai,et al.  StylizedNeRF: Consistent 3D Scene Stylization as Stylized NeRF via 2D-3D Mutual Learning , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  S. Tulyakov,et al.  Control-NeRF: Editable Feature Volumes for Scene Rendering and Manipulation , 2022, 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV).

[24]  Oncel Tuzel,et al.  NeuMan: Neural Human Radiance Field from a Single Video , 2022, ECCV.

[25]  Andreas Geiger,et al.  TensoRF: Tensorial Radiance Fields , 2022, ECCV.

[26]  Russ Tedrake,et al.  Learning Multi-Object Dynamics with Compositional Neural Radiance Fields , 2022, CoRL.

[27]  T. Müller,et al.  Instant neural graphics primitives with a multiresolution hash encoding , 2022, ACM Trans. Graph..

[28]  Benjamin Recht,et al.  Plenoxels: Radiance Fields without Neural Networks , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  S. Fidler,et al.  Extracting Triangular 3D Models, Materials, and Lighting From Images , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Pratul P. Srinivasan,et al.  Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[31]  Hwann-Tzong Chen,et al.  Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  Jonathan T. Barron,et al.  Neural-PIL: Neural Pre-Integrated Lighting for Reflectance Decomposition , 2021, NeurIPS.

[33]  Hujun Bao,et al.  Learning Object-Compositional Neural Radiance Field for Editable Scene Rendering , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[34]  Paul Debevec,et al.  NeRFactor , 2021, ACM Trans. Graph..

[35]  Hung-Yu Tseng,et al.  Stylizing 3D Scene via Implicit Representation and HyperNetwork , 2021, 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV).

[36]  Zhenyi He,et al.  FoV-NeRF: Foveated Neural Radiance Fields for Virtual Reality , 2021, IEEE Transactions on Visualization and Computer Graphics.

[37]  Stefan Leutenegger,et al.  In-Place Scene Labelling and Understanding with Implicit Scene Representation , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[38]  Ren Ng,et al.  PlenOctrees for Real-time Rendering of Neural Radiance Fields , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[39]  Jonathan T. Barron,et al.  NeRD: Neural Reflectance Decomposition from Image Collections , 2020, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[40]  Jonathan T. Barron,et al.  NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis , 2020, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[41]  Pratul P. Srinivasan,et al.  NeRF , 2020, ECCV.

[42]  G. Drettakis,et al.  NeRFshop , 2023, Proc. ACM Comput. Graph. Interact. Tech..