Spatiotemporal reservoir resampling for real-time ray tracing with dynamic direct lighting

Efficiently rendering direct lighting from millions of dynamic light sources using Monte Carlo integration remains a challenging problem, even for off-line rendering systems. We introduce a new algorithm---ReSTIR---that renders such lighting interactively, at high quality, and without needing to maintain complex data structures. We repeatedly resample a set of candidate light samples and apply further spatial and temporal resampling to leverage information from relevant nearby samples. We derive an unbiased Monte Carlo estimator for this approach, and show that it achieves equal-error 6×-60× faster than state-of-the-art methods. A biased estimator reduces noise further and is 35×-65× faster, at the cost of some energy loss. We implemented our approach on the GPU, rendering complex scenes containing up to 3.4 million dynamic, emissive triangles in under 50 ms per frame while tracing at most 8 rays per pixel.

[1]  J. Zára,et al.  Making radiance and irradiance caching practical: adaptive caching and neighbor clamping , 2006, EGSR '06.

[2]  Jaakko Lehtinen,et al.  Anisotropic Gaussian mutations for metropolis light transport through Hessian-Hamiltonian dynamics , 2015, ACM Trans. Graph..

[3]  Yu-Chi Lai,et al.  Photorealistic Image Rendering with Population Monte Carlo Energy Redistribution , 2007, Rendering Techniques.

[4]  Adam Arbree,et al.  To appear in the ACM SIGGRAPH conference proceedings Lightcuts: A Scalable Approach to Illumination , 2022 .

[5]  K.,et al.  A MONTE CARLO STUDY OF SOME RATIO ESTIMATORS , 2022 .

[6]  Alessandro Foi,et al.  Blockwise Multi-Order Feature Regression for Real-Time Path-Tracing Reconstruction , 2019, ACM Trans. Graph..

[7]  Kenny Mitchell,et al.  Adaptive rendering with linear predictions , 2015, ACM Trans. Graph..

[8]  Xi Deng,et al.  Photon surfaces for robust, unbiased volumetric density estimation , 2019, ACM Trans. Graph..

[9]  Anjul Patney,et al.  Spatiotemporal variance-guided filtering: real-time reconstruction for path-traced global illumination , 2017, High Performance Graphics.

[10]  Alejandro Conty Estevez,et al.  Importance sampling of many lights with adaptive tree splitting , 2017, SIGGRAPH Talks.

[11]  Wolfgang Heidrich,et al.  Bidirectional importance sampling for direct illumination , 2005, EGSR '05.

[12]  DachsbacherCarsten,et al.  Scalable Realistic Rendering with Many-Light Methods , 2014 .

[13]  Paul S. Heckbert,et al.  Irradiance gradients , 2008, SIGGRAPH '08.

[14]  Pradeep Sen,et al.  A machine learning approach for filtering Monte Carlo noise , 2015, ACM Trans. Graph..

[15]  Hendrik P. A. Lensch,et al.  Edge-avoiding À-Trous wavelet transform for fast global illumination filtering , 2010, HPG '10.

[16]  Frédo Durand,et al.  Eurographics Symposium on Rendering 2015 Probabilistic Connections for Bidirectional Path Tracing Bidirectional Path Tracing Probabilistic Connections for Bidirectional Path Tracing , 2022 .

[17]  Pablo Bauszat,et al.  Gradient-domain path reusing , 2017, ACM Trans. Graph..

[18]  Leonidas J. Guibas,et al.  Metropolis light transport , 1997, SIGGRAPH.

[19]  Marcus A. Magnor,et al.  Eurographics Symposium on Rendering 2011 Guided Image Filtering for Interactive High-quality Global Illumination , 2022 .

[20]  Markus H. Gross,et al.  Practical Path Guiding for Efficient Light‐Transport Simulation , 2017, Comput. Graph. Forum.

[21]  Matthias Zwicker,et al.  Radiance caching for participating media , 2008, TOGS.

[22]  ZwickerMatthias,et al.  Adaptive sampling and reconstruction using greedy error minimization , 2011 .

[23]  Jaakko Lehtinen,et al.  Gradient-domain metropolis light transport , 2013, ACM Trans. Graph..

[24]  Pavlos S. Efraimidis,et al.  Weighted Random Sampling over Data Streams , 2010, Algorithms, Probability, Networks, and Games.

[25]  Mateu Sbert,et al.  Accelerating Path Tracing by Re-Using Paths , 2002, Rendering Techniques.

[26]  Bochang Moon,et al.  Adaptive Rendering Based on Weighted Local Regression , 2014, ACM Trans. Graph..

[27]  Cem Yuksel,et al.  Real-Time Rendering with Lighting Grid Hierarchy , 2019, PACMCGIT.

[28]  Laurent Belcour,et al.  Distributing Monte Carlo Errors as a Blue Noise in Screen Space by Permuting Pixel Seeds Between Frames , 2019, Comput. Graph. Forum.

[29]  Henrik Wann Jensen,et al.  Global Illumination using Photon Maps , 1996, Rendering Techniques.

[30]  Petrik Clarberg,et al.  Dynamic many-light sampling for real-time ray tracing , 2019, High Performance Graphics.

[31]  Donald P. Greenberg,et al.  Accurate direct illumination using iterative adaptive sampling , 2006, IEEE Transactions on Visualization and Computer Graphics.

[32]  Jerome Spanier,et al.  Quasi-Random Methods for Estimating Integrals Using Relatively Small Samples , 1994, SIAM Rev..

[33]  Tobias Ritschel,et al.  On-line learning of parametric mixture models for light transport simulation , 2014, ACM Trans. Graph..

[34]  Wolfgang Heidrich,et al.  Sequential Sampling for Dynamic Environment Map Illumination , 2022 .

[35]  Bernard Péroche,et al.  Bidirectional instant radiosity , 2006, EGSR '06.

[36]  Paul G. Spirakis,et al.  Weighted random sampling with a reservoir , 2006, Inf. Process. Lett..

[37]  Li Fei-Fei,et al.  Crowdsourcing in Computer Vision , 2016, Found. Trends Comput. Graph. Vis..

[38]  Roberto Manduchi,et al.  Bilateral filtering for gray and color images , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[39]  Remarks on a Monte Carlo integration method , 1964 .

[40]  Werner Purgathofer,et al.  Importance sampling with hemispherical particle footprints , 2002, SCCG '02.

[41]  Leonidas J. Guibas,et al.  Bidirectional Estimators for Light Transport , 1995 .

[42]  Gregory J. Ward,et al.  A ray tracing solution for diffuse interreflection , 2008, SIGGRAPH '08.

[43]  Toshiya Hachisuka,et al.  Multiplexed metropolis light transport , 2014, ACM Trans. Graph..

[44]  Andreas Dietrich,et al.  OptiX: a general purpose ray tracing engine , 2010, SIGGRAPH 2010.

[45]  Mark Meyer,et al.  Denoising with kernel prediction and asymmetric loss functions , 2018, ACM Trans. Graph..

[46]  Kenny Mitchell,et al.  Nonlinearly Weighted First‐order Regression for Denoising Monte Carlo Renderings , 2016, Comput. Graph. Forum.

[47]  Timo Aila,et al.  Interactive reconstruction of Monte Carlo image sequences using a recurrent denoising autoencoder , 2017, ACM Trans. Graph..

[48]  Matthias Zwicker,et al.  Robust Denoising using Feature and Color Information , 2013, Comput. Graph. Forum.

[49]  Morgan McGuire,et al.  An efficient denoising algorithm for global illumination , 2017, High Performance Graphics.

[50]  Parris K. Egbert,et al.  Importance resampling for global illumination , 2005, EGSR '05.

[51]  Thomas G. Dietterich What is machine learning? , 2020, Archives of Disease in Childhood.

[52]  Jean-Michel Morel,et al.  A Review of Image Denoising Algorithms, with a New One , 2005, Multiscale Model. Simul..

[53]  Henrik Wann Jensen,et al.  Practical Hessian-based error control for irradiance caching , 2012, ACM Trans. Graph..

[54]  Yusuke Tokuyoshi,et al.  Hierarchical russian roulette for vertex connections , 2019, ACM Trans. Graph..

[55]  A. J. Walker New fast method for generating discrete random numbers with arbitrary frequency distributions , 1974 .

[56]  Mateu Sbert,et al.  A New Way to Re-using Paths , 2007, ICCSA.

[57]  H. Jensen Realistic Image Synthesis Using Photon Mapping , 2001 .

[58]  M. Chao A general purpose unequal probability sampling plan , 1982 .

[59]  Markus H. Gross,et al.  Spatio‐temporal geometry fusion for multiple hybrid cameras using moving least squares surfaces , 2014, Comput. Graph. Forum.

[60]  Jaakko Lehtinen,et al.  Recent Advances in Adaptive Sampling and Reconstruction for Monte Carlo Rendering , 2015, Comput. Graph. Forum.

[61]  Robert L. Cook,et al.  Stochastic sampling in computer graphics , 1988, TOGS.

[62]  Jan Novák,et al.  Image-space control variates for rendering , 2016, ACM Trans. Graph..

[63]  H. Hartley,et al.  Unbiased Ratio Estimators , 1954, Nature.

[64]  Jaakko Lehtinen,et al.  Reconstructing the indirect light field for global illumination , 2012, ACM Trans. Graph..

[65]  K. Bala Ray Tracing , 1988 .

[66]  M. J. D. Powell,et al.  Weighted Uniform Sampling — a Monte Carlo Technique for Reducing Variance , 1966 .

[67]  J. Spanier A New Family of Estimators for Random Walk Problems , 1979 .

[68]  Pierre Poulin,et al.  Combinatorial Bidirectional Path‐Tracing for Efficient Hybrid CPU/GPU Rendering , 2011, Comput. Graph. Forum.

[69]  Leif Kobbelt,et al.  Theory, analysis and applications of 2D global illumination , 2012, TOGS.

[70]  Jeffrey Scott Vitter,et al.  Random sampling with a reservoir , 1985, TOMS.

[71]  Peter Shirley,et al.  Introduction to DirectX raytracing , 2018, SIGGRAPH Courses.

[72]  Matthias Zwicker,et al.  Adaptive rendering with non-local means filtering , 2012, ACM Trans. Graph..

[73]  David K. McAllister,et al.  OptiX: a general purpose ray tracing engine , 2010, ACM Trans. Graph..

[74]  Mateu Sbert,et al.  Efficient reuse of paths for random walk radiosity , 2008, Comput. Graph..

[75]  Fabio Pellacini,et al.  LightSlice: matrix slice sampling for the many-lights problem , 2011, ACM Trans. Graph..

[76]  YukselCem,et al.  Real-Time Stochastic Lightcuts , 2020 .

[77]  Alexander Keller,et al.  Massively Parallel Path Space Filtering , 2019, MCQMC.

[78]  Matthias Zwicker,et al.  Adaptive sampling and reconstruction using greedy error minimization , 2011, ACM Trans. Graph..

[79]  Derek Nowrouzezahrai,et al.  A comprehensive theory of volumetric radiance estimation using photon points and beams , 2011, TOGS.

[80]  Petr Vévoda,et al.  Bayesian online regression for adaptive direct illumination sampling , 2018, ACM Trans. Graph..

[81]  Luca Fascione,et al.  Path tracing in production - part 2: making movies , 2017, SIGGRAPH Courses.

[82]  Jean-Claude Iehl,et al.  A low-discrepancy sampler that distributes monte carlo errors as a blue noise in screen space , 2019, SIGGRAPH Talks.

[83]  Sumanta N. Pattanaik,et al.  Radiance caching for efficient global illumination computation , 2008, IEEE Transactions on Visualization and Computer Graphics.

[84]  Matthias Zwicker,et al.  Multidimensional adaptive sampling and reconstruction for ray tracing , 2008, ACM Trans. Graph..

[85]  Cem Yuksel,et al.  Stochastic lightcuts , 2019, High Performance Graphics.

[86]  Matthias Zwicker,et al.  Irradiance Gradients in the Presence of Participating Media and Occlusions , 2008, Comput. Graph. Forum.

[87]  Donald B. Rubin,et al.  Comment : A noniterative sampling/importance resampling alternative to the data augmentation algorithm for creating a few imputations when fractions of missing information are modest : The SIR Algorithm , 1987 .

[88]  Chris Wyman,et al.  Exploring and expanding the continuum of OIT algorithms , 2016, High Performance Graphics.

[89]  Peter Shirley,et al.  Monte Carlo techniques for direct lighting calculations , 1996, TOGS.

[90]  Philipp Slusallek,et al.  Combining global and local virtual lights for detailed glossy illumination , 2010, SIGGRAPH 2010.

[91]  Morgan McGuire,et al.  Combining analytic direct illumination and stochastic shadows , 2018, I3D.

[92]  Philipp Slusallek,et al.  Optimal multiple importance sampling , 2019, ACM Trans. Graph..

[93]  K. Bala,et al.  Lightcuts: a scalable approach to illumination , 2005, SIGGRAPH 2005.

[94]  Gregory J. Ward,et al.  Adaptive Shadow Testing for Ray Tracing , 1994 .

[95]  BENEDIKT BITTERLI,et al.  Selectively metropolised Monte Carlo light transport simulation , 2019, ACM Trans. Graph..

[96]  Matthias Zwicker,et al.  The Beam Radiance Estimate for Volumetric Photon Mapping , 2008, SIGGRAPH '08.

[97]  Ingo Wald,et al.  Sequential Monte Carlo Adaptation in Low-Anisotropy Participating Media , 2008, Comput. Graph. Forum.

[98]  F. Durand,et al.  Temporal light field reconstruction for rendering distribution effects , 2011, ACM Trans. Graph..

[99]  Jacopo Pantaleoni,et al.  Matrix Bidirectional Path Tracing , 2018, EGSR.

[100]  Cem Yuksel,et al.  Real-Time Stochastic Lightcuts , 2020, Proc. ACM Comput. Graph. Interact. Tech..

[101]  Adam Arbree,et al.  Scalable Realistic Rendering with Many‐Light Methods , 2014, Eurographics.

[102]  Wolfgang Heidrich,et al.  Bidirectional importance sampling for illumination from environment maps , 2004, SIGGRAPH '04.

[103]  M. R. Mickey,et al.  Some Finite Population Unbiased Ratio and Regression Estimators , 1959 .

[104]  Per H. Christensen,et al.  The Path to Path-Traced Movies , 2016, Found. Trends Comput. Graph. Vis..

[105]  Reginald Gerald. Worthley Unbiased ratio-type estimators , 1967 .

[106]  Leonidas J. Guibas,et al.  Optimally combining sampling techniques for Monte Carlo rendering , 1995, SIGGRAPH.

[107]  Henrik Wann Jensen,et al.  Importance Driven Path Tracing using the Photon Map , 1995, Rendering Techniques.

[108]  Steven McDonagh,et al.  Adaptive polynomial rendering , 2016, ACM Trans. Graph..

[109]  Alexander Keller,et al.  Instant radiosity , 1997, SIGGRAPH.

[110]  Adam Arbree,et al.  Multidimensional lightcuts , 2006, ACM Trans. Graph..

[111]  Mateu Sbert,et al.  Real‐time Light Animation , 2004, Comput. Graph. Forum.

[112]  Csaba Kelemen,et al.  Simple and Robust Mutation Strategy for Metropolis Light Transport Algorithm , 2001 .

[113]  Justin Talbot,et al.  Energy redistribution path tracing , 2005, ACM Trans. Graph..

[114]  Yves D. Willems,et al.  Bi-directional path tracing , 1993 .

[115]  Toshiya Hachisuka,et al.  Geometry-aware metropolis light transport , 2018, ACM Trans. Graph..

[116]  Carsten Dachsbacher,et al.  Gradient Estimation for Real-time Adaptive Temporal Filtering , 2018, PACMCGIT.

[117]  Mateu Sbert,et al.  Weighted Importance Sampling Techniques for Monte Carlo Radiosity , 2000, Rendering Techniques.

[118]  Ulf Assarsson,et al.  Tiled Shading , 2011, J. Graphics, GPU, & Game Tools.

[119]  Marcos Fajardo,et al.  Blue-noise dithered sampling , 2016, SIGGRAPH Talks.

[120]  Jian Sun,et al.  Guided Image Filtering , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.