Temporally Adaptive Shading Reuse for Real-Time Rendering and Virtual Reality

Temporal coherence has the potential to enable a huge reduction of shading costs in rendering. Existing techniques focus either only on spatial shading reuse or cannot adaptively choose temporal shading frequencies. We find that temporal shading reuse is possible for extended periods of time for a majority of samples, and we show under which circumstances users perceive temporal artifacts. Our analysis implies that we can approximate shading gradients to efficiently determine when and how long shading can be reused. Whereas visibility usually stays temporally coherent from frame to frame for more than 90%, we find that even in heavily animated game scenes with advanced shading, typically more than 50% of shading is also temporally coherent. To exploit this potential, we introduce a temporally adaptive shading framework and apply it to two real-time methods. Its application saves more than 57% of the shader invocations, reducing overall rendering times up to in virtual reality applications without a noticeable loss in visual quality. Overall, our work shows that there is significantly more potential for shading reuse than currently exploited.

[1]  Desney S. Tan,et al.  Foveated 3D graphics , 2012, ACM Trans. Graph..

[2]  Hans-Peter Seidel,et al.  Spatio-temporal upsampling on the GPU , 2010, I3D '10.

[3]  Tomas Akenine-Möller,et al.  Adaptive texture space shading for stochastic rendering , 2014, Comput. Graph. Forum.

[4]  Sebastian Möller,et al.  An Evaluation of Video Quality Assessment Metrics for Passive Gaming Video Streaming , 2018, PV@MMSys.

[5]  Frédo Durand,et al.  A frequency analysis of light transport , 2005, SIGGRAPH '05.

[6]  Eric B. Lum,et al.  Visually Lossless Content and Motion Adaptive Shading in Games , 2019, PACMCGIT.

[7]  K. Bala,et al.  Lightcuts: a scalable approach to illumination , 2005, SIGGRAPH 2005.

[8]  Ravi Ramamoorthi,et al.  A first-order analysis of lighting, shading, and shadows , 2007, TOGS.

[9]  Thomas Neff,et al.  Shading atlas streaming , 2018, ACM Trans. Graph..

[10]  Rafal Mantiuk,et al.  Comparison of Four Subjective Methods for Image Quality Assessment , 2012, Comput. Graph. Forum.

[11]  Eric Enderton,et al.  Real-time stochastic rasterization on conventional GPU architectures , 2010, HPG '10.

[12]  Jaakko Lehtinen,et al.  Differentiable Monte Carlo ray tracing through edge sampling , 2018, ACM Trans. Graph..

[13]  William R. Mark,et al.  A lazy object-space shading architecture with decoupled sampling , 2010, HPG '10.

[14]  E. Adelson Lightness Perception and Lightness Illusions , 1999 .

[15]  Guillaume Abadie,et al.  Advances in real-time rendering in games , 2018, ACM SIGGRAPH 2018 Courses.

[16]  K. Bala,et al.  Lightcuts: a scalable approach to illumination , 2005, SIGGRAPH '05.

[17]  Tatsuya Harada,et al.  Neural 3D Mesh Renderer , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[18]  Carsten Dachsbacher,et al.  Gradient Estimation for Real-time Adaptive Temporal Filtering , 2018, PACMCGIT.

[19]  Hans-Peter Seidel,et al.  Perceptually‐motivated Real‐time Temporal Upsampling of 3D Content for High‐refresh‐rate Displays , 2010, Comput. Graph. Forum.

[20]  George Drettakis,et al.  Interactive Rendering using the Render Cache , 1999, Rendering Techniques.

[21]  Jason Lawrence,et al.  An improved shading cache for modern GPUs , 2008, GH '08.

[22]  Jason Lawrence,et al.  Accelerating real-time shading with reverse reprojection caching , 2007, GH '07.

[23]  Margaret H. Pinson,et al.  A new standardized method for objectively measuring video quality , 2004, IEEE Transactions on Broadcasting.

[24]  Donald P. Greenberg,et al.  Interactive global illumination in dynamic scenes , 2002, SIGGRAPH.

[25]  H. Blackwell Luminance Difference Thresholds , 1972 .

[26]  Ersin Yumer,et al.  Material Editing Using a Physically Based Rendering Network , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[27]  Mikhail Okunev,et al.  DeepFovea , 2019, ACM Trans. Graph..

[28]  J. M. P. van Waveren,et al.  The asynchronous time warp for virtual reality on consumer hardware , 2016, VRST.

[29]  Lei Yang,et al.  Image-based bidirectional scene reprojection , 2011, ACM Trans. Graph..

[30]  Zhou Wang,et al.  Information Content Weighting for Perceptual Image Quality Assessment , 2011, IEEE Transactions on Image Processing.

[31]  Khaled El-Maleh,et al.  Perceptual Temporal Quality Metric for Compressed Video , 2007, IEEE Transactions on Multimedia.

[32]  HeidrichWolfgang,et al.  HDR-VDP-2 , 2011 .

[33]  Robert L. Cook,et al.  The Reyes image rendering architecture , 1987, SIGGRAPH.

[34]  Yan Gu,et al.  Extending the graphics pipeline with adaptive, multi-rate shading , 2014, ACM Trans. Graph..

[35]  James Hu,et al.  DVQ: A digital video quality metric based on human vision , 2001 .

[36]  Hans-Peter Seidel,et al.  Towards a Quality Metric for Dense Light Fields , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  Kenny Mitchell,et al.  Pixel History Linear Models for Real‐Time Temporal Filtering , 2016, Comput. Graph. Forum.

[38]  Jason Lawrence,et al.  Amortized supersampling , 2009, ACM Trans. Graph..

[39]  J. C. Yang,et al.  Texel Shading , 2016, Eurographics.

[40]  Tomas Akenine-Möller,et al.  AMFS: adaptive multi-frequency shading for future graphics processors , 2014, ACM Trans. Graph..

[41]  Kenny Mitchell,et al.  User, metric, and computational evaluation of foveated rendering methods , 2016, SAP.

[42]  Olivier Verscheure,et al.  Perceptual quality measure using a spatiotemporal model of the human visual system , 1996, Electronic Imaging.

[43]  Lei Yang,et al.  Temporal Coherence Methods in Real‐Time Rendering , 2012, Comput. Graph. Forum.

[44]  Alan C. Bovik,et al.  Motion Tuned Spatio-Temporal Quality Assessment of Natural Videos , 2010, IEEE Transactions on Image Processing.

[45]  Joohwan Kim,et al.  Towards foveated rendering for gaze-tracked virtual reality , 2016, ACM Trans. Graph..

[46]  Tomas Akenine-Möller,et al.  FLIP: A Difference Evaluator for Alternating Images , 2020, Proc. ACM Comput. Graph. Interact. Tech..

[47]  Christophe Schlick,et al.  An Inexpensive BRDF Model for Physically‐based Rendering , 1994, Comput. Graph. Forum.

[48]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[49]  Aaron E. Lefohn,et al.  Coarse Pixel Shading , 2014, High Performance Graphics.

[50]  Michael J. Black,et al.  OpenDR: An Approximate Differentiable Renderer , 2014, ECCV.

[51]  Wolfgang Heidrich,et al.  HDR-VDP-2: a calibrated visual metric for visibility and quality predictions in all luminance conditions , 2011, SIGGRAPH 2011.

[52]  M. Luo,et al.  The development of the CIE 2000 Colour Difference Formula , 2001 .

[53]  Benjamin Watson,et al.  Adaptive frameless rendering , 2005, EGSR '05.

[54]  Jaakko Lehtinen,et al.  Decoupled sampling for graphics pipelines , 2011, TOGS.

[55]  Yong He,et al.  A system for rapid exploration of shader optimization choices , 2016, ACM Trans. Graph..