Image-based bidirectional scene reprojection

We introduce a method for increasing the framerate of real-time rendering applications. Whereas many existing temporal upsampling strategies only reuse information from previous frames, our bidirectional technique reconstructs intermediate frames from a pair of consecutive rendered frames. This significantly improves the accuracy and efficiency of data reuse since very few pixels are simultaneously occluded in both frames. We present two versions of this basic algorithm. The first is appropriate for fill-bound scenes as it limits the number of expensive shading calculations, but involves rasterization of scene geometry at each intermediate frame. The second version, our more significant contribution, reduces both shading and geometry computations by performing reprojection using only image-based buffers. It warps and combines the adjacent rendered frames using an efficient iterative search on their stored scene depth and flow. Bidirectional reprojection introduces a small amount of lag. We perform a user study to investigate this lag, and find that its effect is minor. We demonstrate substantial performance improvements (3--4x) for a variety of applications, including vertex-bound and fill-bound scenes, multi-pass effects, and motion blur.

[1]  Henry Fuchs,et al.  Frameless rendering: double buffering considered harmful , 1994, SIGGRAPH.

[2]  Hans-Peter Seidel,et al.  An efficient spatio-temporal architecture for animation rendering , 2003, SIGGRAPH '03.

[3]  Hans-Peter Seidel,et al.  Walkthroughs with Corrective Texturing , 2000, Rendering Techniques.

[4]  Michael Wimmer,et al.  High‐Quality Screen‐Space Ambient Occlusion using Temporal Coherence , 2010, Comput. Graph. Forum.

[5]  Jason Lawrence,et al.  Accelerating real-time shading with reverse reprojection caching , 2007, GH '07.

[6]  Takeo Kanade,et al.  Spatio-Temporal View Interpolation , 2002, Rendering Techniques.

[7]  Michael Wimmer,et al.  Real-Time Soft Shadows Using Temporal Coherence , 2009, ISVC.

[8]  Lance Williams,et al.  View Interpolation for Image Synthesis , 1993, SIGGRAPH.

[9]  Rui Wang,et al.  Symposium on Interactive 3D Graphics and Games , 2011, i3D 2011.

[10]  Jason Lawrence,et al.  Automated reprojection-based pixel shader optimization , 2008, SIGGRAPH Asia '08.

[11]  Rui Wang,et al.  GPU_Accelerated Render Cache , 2012 .

[12]  Jason Lawrence,et al.  An improved shading cache for modern GPUs , 2008, GH '08.

[13]  Maryann Simmons,et al.  The holodeck ray cache: an interactive rendering system for global illumination in nondiffuse environments , 1999, TOGS.

[14]  Benjamin Watson,et al.  Adaptive frameless rendering , 2005, EGSR '05.

[15]  Anita Sellent,et al.  Floating Textures , 2008, Comput. Graph. Forum.

[16]  Wojciech Matusik,et al.  Moving gradients: a path-based method for plausible image interpolation , 2009, ACM Trans. Graph..

[17]  John Hart,et al.  ACM Transactions on Graphics , 2004, SIGGRAPH 2004.

[18]  Hans-Peter Seidel,et al.  Spatio-temporal upsampling on the GPU , 2010, I3D '10.

[19]  Sig Badt,et al.  Two algorithms for taking advantage of temporal coherence in ray tracing , 1988, The Visual Computer.

[20]  George Drettakis,et al.  Interactive Rendering using the Render Cache , 1999, Rendering Techniques.

[21]  Larry F. Hodges,et al.  Generating exact ray-traced animation frames by reprojection , 1995, IEEE Computer Graphics and Applications.

[22]  Marcus A. Magnor,et al.  View and Time Interpolation in Image Space , 2008, Comput. Graph. Forum.

[23]  Jason Lawrence,et al.  Amortized supersampling , 2009, ACM Trans. Graph..

[24]  Gary J. Sullivan,et al.  Video Compression - From Concepts to the H.264/AVC Standard , 2005, Proceedings of the IEEE.

[25]  George Drettakis,et al.  Enhancing and Optimizing the Render Cache , 2002, Rendering Techniques.

[26]  Carlo H. Séquin,et al.  Tapestry: A Dynamic Mesh-based Display Representation for Interactive Rendering , 2000, Rendering Techniques.

[27]  Lei Yang,et al.  Temporal Coherence Methods in Real‐Time Rendering , 2012, Comput. Graph. Forum.

[28]  Ajay Luthra,et al.  Overview of the H.264/AVC video coding standard , 2003, IEEE Trans. Circuits Syst. Video Technol..

[29]  Andrew W. Fitzgibbon,et al.  Image-Based Rendering Using Image-Based Priors , 2005, International Journal of Computer Vision.

[30]  Sumanta N. Pattanaik,et al.  Radiance cache splatting: a GPU-friendly global illumination algorithm , 2005, EGSR '05.

[31]  Steven M. Seitz,et al.  View morphing , 1996, SIGGRAPH.

[32]  Robert L. Cook,et al.  The Reyes image rendering architecture , 1987, SIGGRAPH.

[33]  Benjamin Watson,et al.  Interruptible rendering , 2002, SIGGRAPH '02.

[34]  Lei Yang,et al.  A Survey on Temporal Coherence Methods in Real-Time Rendering , 2011, Eurographics.

[35]  Leonard McMillan,et al.  Plenoptic Modeling: An Image-Based Rendering System , 2023 .

[36]  Thaddeus Beier,et al.  Feature-based image metamorphosis , 1992, SIGGRAPH.

[37]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[38]  Marcus A. Magnor,et al.  Perception-motivated interpolation of image sequences , 2008, TAP.

[39]  Hans-Peter Seidel,et al.  Scalable Remote Rendering with Depth and Motion‐flow Augmented Streaming , 2011, Comput. Graph. Forum.

[40]  Dmitry Andreev,et al.  Real-time frame rate up-conversion for video games: or how to get from 30 to 60 fps for "free" , 2010, SIGGRAPH '10.

[41]  Hans-Peter Seidel,et al.  Adaptive Image-space Stereo View Synthesis , 2010, VMV.

[42]  Hans-Peter Seidel,et al.  Perceptually‐motivated Real‐time Temporal Upsampling of 3D Content for High‐refresh‐rate Displays , 2010, Comput. Graph. Forum.

[43]  Leonard McMillan,et al.  Post-rendering 3D warping , 1997, SI3D.

[44]  Hans-Peter Seidel,et al.  Exploiting temporal coherence in final gathering for dynamic scenes , 2004, Proceedings Computer Graphics International, 2004..

[45]  Seth J. Teller,et al.  Radiance interpolants for accelerated bounded-error ray tracing , 1999, TOGS.

[46]  Michael Wimmer,et al.  Pixel-Correct Shadow Maps with Temporal Reprojection and Shadow Test Confidence , 2007, Rendering Techniques.