Foveated Neural Radiance Fields for Real-Time and Egocentric Virtual Reality

Fig. 1. Illustration of our gaze-contingent neural synthesis method. (a) Visualization of our active-viewing-tailored egocentric neural scene representation. (b) Our method allows for extremely low data storage of high-quality 3D scene assets and achieves high perceptual image quality and low latency for interactive virtual reality. Here, we compare our method with the state-of-the-art alternative neural synthesis [Mildenhall et al. 2020] and foveated rendering solutions [Patney et al. 2016; Perry and Geisler 2002] and demonstrate that our method significantly reduces more than 99% time consumption (from 9s to 20ms) and data storage (from 100MB mesh to 0.5MB neural model) for first-person immersive viewing. The orange circle indicates the viewer’s gaze position. We also show zoomed-in views of the foveal images generated by various methods in (c) and show that our method is superior compared to existing methods in achieving high perceptual image fidelity.