Recent Advances in Adaptive Sampling and Reconstruction for Monte Carlo Rendering

Monte Carlo integration is firmly established as the basis for most practical realistic image synthesis algorithms because of its flexibility and generality. However, the visual quality of rendered images often suffers from estimator variance, which appears as visually distracting noise. Adaptive sampling and reconstruction algorithms reduce variance by controlling the sampling density and aggregating samples in a reconstruction step, possibly over large image regions. In this paper we survey recent advances in this area. We distinguish between “a priori” methods that analyze the light transport equations and derive sampling rates and reconstruction filters from this analysis, and “a posteriori” methods that apply statistical techniques to sets of samples to drive the adaptive sampling and reconstruction process. They typically estimate the errors of several reconstruction filters, and select the best filter locally to minimize error. We discuss advantages and disadvantages of recent state‐of‐the‐art techniques, and provide visual and quantitative comparisons. Some of these techniques are proving useful in real‐world applications, and we aim to provide an overview for practitioners and researchers to assess these approaches. In addition, we discuss directions for potential further improvements.

[1]  F. Durand,et al.  Flash photography enhancement via intrinsic relighting , 2004, ACM Trans. Graph..

[2]  Jean-Michel Morel,et al.  A Review of Image Denoising Algorithms, with a New One , 2005, Multiscale Model. Simul..

[3]  Michael Elad,et al.  Image Denoising Via Sparse and Redundant Representations Over Learned Dictionaries , 2006, IEEE Transactions on Image Processing.

[4]  Alexander Keller,et al.  Path space filtering , 2014, SIGGRAPH Talks.

[5]  Sumanta N. Pattanaik,et al.  Radiance caching for efficient global illumination computation , 2008, IEEE Transactions on Visualization and Computer Graphics.

[6]  Leif Kobbelt,et al.  Theory, analysis and applications of 2D global illumination , 2012, TOGS.

[7]  Yung-Yu Chuang,et al.  SURE-based optimization for adaptive sampling and reconstruction , 2012, ACM Trans. Graph..

[8]  Paul S. Heckbert,et al.  Fundamentals of Texture Mapping and Image Warping , 1989 .

[9]  Frédo Durand,et al.  A frequency analysis of light transport , 2005, SIGGRAPH '05.

[10]  Holly E. Rushmeier,et al.  Energy preserving non-linear filters , 1994, SIGGRAPH.

[11]  Cyril Soler,et al.  A Local Frequency Analysis of Light Scattering and Absorption , 2014, ACM Trans. Graph..

[12]  Gary W. Meyer,et al.  A perceptually based adaptive sampling algorithm , 1998, SIGGRAPH.

[13]  Matthias Zwicker,et al.  Multidimensional adaptive sampling and reconstruction for ray tracing , 2008, ACM Trans. Graph..

[14]  Kadi Bouatouch,et al.  Radiance caching for efficient global illumination computation , 2005 .

[15]  Matthias Zwicker,et al.  Irradiance Gradients in the Presence of Participating Media and Occlusions , 2008, Comput. Graph. Forum.

[16]  Ravi Ramamoorthi,et al.  A first-order analysis of lighting, shading, and shadows , 2007, TOGS.

[17]  Ravi Ramamoorthi,et al.  Axis-aligned filtering for interactive sampled soft shadows , 2012, ACM Trans. Graph..

[18]  Karthikeyan Vaidyanathan,et al.  Layered Light Field Reconstruction for Defocus Blur , 2015, TOGS.

[19]  Soheil Darabi,et al.  Compressive Rendering: A Rendering Application of Compressed Sensing , 2011, IEEE Transactions on Visualization and Computer Graphics.

[20]  A. W. M. van den Enden,et al.  Discrete Time Signal Processing , 1989 .

[21]  Frédo Durand,et al.  Fourier depth of field , 2009, TOGS.

[22]  F. Durand,et al.  Temporal light field reconstruction for rendering distribution effects , 2011, ACM Trans. Graph..

[23]  Toshiya Hachisuka,et al.  Robust Image Denoising Using a Virtual Flash Image for Monte Carlo Ray Tracing , 2013, Comput. Graph. Forum.

[24]  Hendrik P. A. Lensch,et al.  Edge-avoiding À-Trous wavelet transform for fast global illumination filtering , 2010, HPG '10.

[25]  Don P. Mitchell,et al.  Generating antialiased images at low sampling densities , 1987, SIGGRAPH.

[26]  Frédo Durand,et al.  5D Covariance tracing for efficient defocus and motion blur , 2013, TOGS.

[27]  Jaakko Lehtinen,et al.  Reconstructing the indirect light field for global illumination , 2012, ACM Trans. Graph..

[28]  Luke Goddard,et al.  Silencing the noise on Elysium , 2014, SIGGRAPH Talks.

[29]  Richard Peter Weistroffer,et al.  Multidimensional adaptive sampling and reconstruction for ray tracing , 2008, SIGGRAPH 2008.

[30]  Kartic Subr,et al.  Interactive Rendering of Acquired Materials on Dynamic Geometry Using Frequency Analysis , 2013, IEEE Transactions on Visualization and Computer Graphics.

[31]  I. Johnstone,et al.  Ideal spatial adaptation by wavelet shrinkage , 1994 .

[32]  Adam Arbree,et al.  To appear in the ACM SIGGRAPH conference proceedings Lightcuts: A Scalable Approach to Illumination , 2022 .

[33]  Michael D. McCool,et al.  Anisotropic diffusion for Monte Carlo noise reduction , 1999, TOGS.

[34]  Baining Guo,et al.  Progressive radiance evaluation using directional coherence maps , 1998, SIGGRAPH.

[35]  Marcus A. Magnor,et al.  Eurographics Symposium on Rendering 2011 Guided Image Filtering for Interactive High-quality Global Illumination , 2022 .

[36]  Michael F. Cohen,et al.  Digital photography with flash and no-flash image pairs , 2004, ACM Trans. Graph..

[37]  Frédo Durand,et al.  Factored axis-aligned filtering for rendering multiple distribution effects , 2014, ACM Trans. Graph..

[38]  Tim Weyrich,et al.  Density‐based Outlier Rejection in Monte Carlo Rendering , 2010, Comput. Graph. Forum.

[39]  Carsten Dachsbacher,et al.  Efficient Monte Carlo rendering with realistic lenses , 2014, Comput. Graph. Forum.

[40]  Harry Shum,et al.  Plenoptic sampling , 2000, SIGGRAPH.

[41]  Jian Sun,et al.  Guided Image Filtering , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[42]  Tomas Akenine-Möller,et al.  Layered Reconstruction for Defocus and Motion Blur , 2014, Comput. Graph. Forum.

[43]  R. Ramamoorthi,et al.  Adaptive wavelet rendering , 2009, SIGGRAPH 2009.

[44]  Dimitri Van De Ville,et al.  SURE-Based Non-Local Means , 2009, IEEE Signal Processing Letters.

[45]  Jacopo Pantaleoni,et al.  A path space extension for robust light transport simulation , 2012, ACM Trans. Graph..

[46]  Frédo Durand,et al.  Practical filtering for efficient ray-traced directional occlusion , 2011, ACM Trans. Graph..

[47]  Mikio Shinya Spatial anti-aliasing for animation sequences with spatio-temporal filtering , 1993, SIGGRAPH.

[48]  J. Painter,et al.  Antialiased ray tracing by adaptive progressive refinement , 1989, SIGGRAPH.

[49]  Matthias Zwicker,et al.  Adaptive sampling and reconstruction using greedy error minimization , 2011, ACM Trans. Graph..

[50]  Adam Arbree,et al.  Multidimensional lightcuts , 2006, ACM Trans. Graph..

[51]  Matthias Zwicker,et al.  Robust Denoising using Feature and Color Information , 2013, Comput. Graph. Forum.

[52]  Ruifeng Xu,et al.  A novel Monte Carlo noise reduction operator , 2005, IEEE Computer Graphics and Applications.

[53]  James T. Kajiya,et al.  The rendering equation , 1986, SIGGRAPH.

[54]  Frédo Durand,et al.  Antialiasing for automultiscopic 3D displays , 2006, EGSR '06.

[55]  Frédo Durand,et al.  Frequency analysis and sheared filtering for shadow light fields of complex occluders , 2011, TOGS.

[56]  Pradeep Sen,et al.  Removing the Noise in Monte Carlo Rendering with General Image Denoising Algorithms , 2013, Comput. Graph. Forum.

[57]  Matthias Zwicker,et al.  Radiance caching for participating media , 2008, TOGS.

[58]  Bochang Moon,et al.  P-RPF: Pixel-Based Random Parameter Filtering for Monte Carlo Rendering , 2013, 2013 International Conference on Computer-Aided Design and Computer Graphics.

[59]  Diego Gutierrez,et al.  A framework for transient rendering , 2014, ACM Trans. Graph..

[60]  Henrik Wann Jensen,et al.  Practical Hessian-based error control for irradiance caching , 2012, ACM Trans. Graph..

[61]  Frédo Durand,et al.  Frequency analysis and sheared reconstruction for rendering motion blur , 2009, ACM Trans. Graph..

[62]  Alessandro Foi,et al.  Image Denoising by Sparse 3-D Transform-Domain Collaborative Filtering , 2007, IEEE Transactions on Image Processing.

[63]  J. Barbič,et al.  Real-time large-deformation substructuring , 2011, SIGGRAPH 2011.

[64]  Yaser Sheikh,et al.  Automatic editing of footage from multiple social cameras , 2014, ACM Trans. Graph..

[65]  Sumanta N. Pattanaik,et al.  Improved radiance gradient computation , 2005, SCCG '05.

[66]  Frédo Durand,et al.  Axis-aligned filtering for interactive physically-based diffuse indirect lighting , 2013, ACM Trans. Graph..

[67]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[68]  H. Jensen Realistic Image Synthesis Using Photon Mapping , 2001 .

[69]  Mark Meyer,et al.  Statistical acceleration for animated global illumination , 2006, SIGGRAPH 2006.

[70]  Philipp Slusallek,et al.  Light transport simulation with vertex connection and merging , 2012, ACM Trans. Graph..

[71]  Don P. Mitchell,et al.  Spectrally optimal sampling for distribution ray tracing , 1991, SIGGRAPH.

[72]  F. Durand,et al.  A frequency analysis of light transport , 2005, ACM Trans. Graph..

[73]  Adolfo Muñoz,et al.  Higher Order Ray Marching , 2014, Comput. Graph. Forum.

[74]  Gregory J. Ward,et al.  A ray tracing solution for diffuse interreflection , 2008, SIGGRAPH '08.

[75]  Greg Humphreys,et al.  Physically Based Rendering: From Theory to Implementation , 2004 .

[76]  Gordon Wetzstein,et al.  Layered 3D: tomographic image synthesis for attenuation-based light field and high dynamic range displays , 2011, ACM Trans. Graph..

[77]  Bochang Moon,et al.  Adaptive Rendering Based on Weighted Local Regression , 2014, ACM Trans. Graph..

[78]  Gordon Wetzstein,et al.  Adaptive image synthesis for compressive displays , 2013, ACM Trans. Graph..

[79]  Donald P. Greenberg,et al.  A perceptually based physical error metric for realistic image synthesis , 1999, SIGGRAPH.

[80]  Frédo Durand,et al.  Frequency analysis and sheared reconstruction for rendering motion blur , 2009, SIGGRAPH 2009.

[81]  Matthias Zwicker,et al.  Adaptive rendering with non-local means filtering , 2012, ACM Trans. Graph..

[82]  Roberto Manduchi,et al.  Bilateral filtering for gray and color images , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[83]  Paul S. Heckbert,et al.  Survey of Texture Mapping , 1986, IEEE Computer Graphics and Applications.

[84]  Jean-Michel Morel,et al.  Boosting monte carlo rendering by ray histogram fusion , 2014, ACM Trans. Graph..

[85]  Paul S. Heckbert,et al.  Irradiance gradients , 2008, SIGGRAPH '08.

[86]  Soheil Darabi,et al.  On filtering the noise from the random parameters in Monte Carlo rendering , 2012, TOGS.

[87]  Daniele Perrone,et al.  Total Variation Blind Deconvolution: The Devil Is in the Details , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[88]  Bernard Péroche,et al.  A Progressive Rendering Algorithm Using an Adaptive Perceptually Based Image Metric , 2004, Comput. Graph. Forum.

[89]  Derek Nowrouzezahrai,et al.  Unifying points, beams, and paths in volumetric light transport simulation , 2014, ACM Trans. Graph..

[90]  Frédo Durand,et al.  4D frequency analysis of computational cameras for depth of field extension , 2009, SIGGRAPH '09.