Interactive reconstruction of Monte Carlo image sequences using a recurrent denoising autoencoder

We describe a machine learning technique for reconstructing image sequences rendered using Monte Carlo methods. Our primary focus is on reconstruction of global illumination with extremely low sampling budgets at interactive rates. Motivated by recent advances in image restoration with deep convolutional networks, we propose a variant of these networks better suited to the class of noise present in Monte Carlo rendering. We allow for much larger pixel neighborhoods to be taken into account, while also improving execution speed by an order of magnitude. Our primary contribution is the addition of recurrent connections to the network in order to drastically improve temporal stability for sequences of sparsely sampled input images. Our method also has the desirable property of automatically modeling relationships based on auxiliary per-pixel input channels, such as depth and normals. We show significantly higher quality results compared to existing methods that run at comparable speeds, and furthermore argue a clear path for making our method run at realtime rates in the near future.

[1]  J. Halton,et al.  Algorithm 247: Radical-inverse quasi-random point sequence , 1964, CACM.

[2]  C. Stein Estimation of the Mean of a Multivariate Normal Distribution , 1981 .

[3]  PAUL J. WERBOS,et al.  Generalization of backpropagation with application to a recurrent gas market model , 1988, Neural Networks.

[4]  Neil Hunt,et al.  The triangle processor and normal vector shader: a VLSI system for high performance graphics , 1988, SIGGRAPH.

[5]  R. Redner,et al.  A note on the use of nonlinear filtering in computer graphics , 1990, IEEE Computer Graphics and Applications.

[6]  Leonidas J. Guibas,et al.  Robust Monte Carlo methods for light transport simulation , 1997 .

[7]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[8]  James T. Kajiya,et al.  The rendering equation , 1998 .

[9]  Michael D. McCool,et al.  Anisotropic diffusion for Monte Carlo noise reduction , 1999, TOGS.

[10]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[11]  Yoshua Bengio,et al.  Extracting and composing robust features with denoising autoencoders , 2008, ICML '08.

[12]  Peter Shirley,et al.  Image space gathering , 2009, High Performance Graphics.

[13]  Frédo Durand,et al.  Frequency analysis and sheared reconstruction for rendering motion blur , 2009, SIGGRAPH 2009.

[14]  Hendrik P. A. Lensch,et al.  Edge-avoiding À-Trous wavelet transform for fast global illumination filtering , 2010, HPG '10.

[15]  David K. McAllister,et al.  OptiX: a general purpose ray tracing engine , 2010, ACM Trans. Graph..

[16]  Hendrik P. A. Lensch,et al.  Edge‐Optimized À‐Trous Wavelets for Local Contrast Enhancement with Robust Denoising , 2011, Comput. Graph. Forum.

[17]  Frédo Durand,et al.  Practical filtering for efficient ray-traced directional occlusion , 2011, ACM Trans. Graph..

[18]  Yoram Bresler,et al.  MR Image Reconstruction From Highly Undersampled k-Space Data by Dictionary Learning , 2011, IEEE Transactions on Medical Imaging.

[19]  Marcus A. Magnor,et al.  Eurographics Symposium on Rendering 2011 Guided Image Filtering for Interactive High-quality Global Illumination , 2022 .

[20]  Frédo Durand,et al.  Frequency analysis and sheared filtering for shadow light fields of complex occluders , 2011, TOGS.

[21]  Manuel Menezes de Oliveira Neto,et al.  Adaptive manifolds for real-time high-dimensional filtering , 2012, ACM Trans. Graph..

[22]  Yung-Yu Chuang,et al.  SURE-based optimization for adaptive sampling and reconstruction , 2012, ACM Trans. Graph..

[23]  Jaakko Lehtinen,et al.  Reconstructing the indirect light field for global illumination , 2012, ACM Trans. Graph..

[24]  Ravi Ramamoorthi,et al.  Axis-aligned filtering for interactive sampled soft shadows , 2012, ACM Trans. Graph..

[25]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[26]  Tapani Raiko,et al.  Deep Learning Made Easier by Linear Transformations in Perceptrons , 2012, AISTATS.

[27]  Soheil Darabi,et al.  On filtering the noise from the random parameters in Monte Carlo rendering , 2012, TOGS.

[28]  Andrew L. Maas Rectifier Nonlinearities Improve Neural Network Acoustic Models , 2013 .

[29]  Matthias Zwicker,et al.  Robust Denoising using Feature and Color Information , 2013, Comput. Graph. Forum.

[30]  Anton Kaplanyan,et al.  Path Space Regularization for Holistic and Robust Light Transport , 2013, Comput. Graph. Forum.

[31]  Jacco Bikker,et al.  The Brigade Renderer: A Path Tracer for Real-Time Games , 2013, Int. J. Comput. Games Technol..

[32]  Karsten Schwenk,et al.  Filtering techniques for low-noise previews of interactive stochastic ray tracing , 2013 .

[33]  Frédo Durand,et al.  Axis-aligned filtering for interactive physically-based diffuse indirect lighting , 2013, ACM Trans. Graph..

[34]  Trevor Darrell,et al.  Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation , 2013, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[35]  Frédo Durand,et al.  Factored axis-aligned filtering for rendering multiple distribution effects , 2014, ACM Trans. Graph..

[36]  Jürgen Schmidhuber,et al.  Deep learning in neural networks: An overview , 2014, Neural Networks.

[37]  Liang Wang,et al.  Bidirectional Recurrent Convolutional Networks for Multi-Frame Super-Resolution , 2015, NIPS.

[38]  Marcus A. Magnor,et al.  Sample‐Based Manifold Filtering for Interactive Global Illumination and Depth of Field , 2015, Comput. Graph. Forum.

[39]  Pradeep Sen,et al.  A machine learning approach for filtering Monte Carlo noise , 2015, ACM Trans. Graph..

[40]  Jaakko Lehtinen,et al.  Recent Advances in Adaptive Sampling and Reconstruction for Monte Carlo Rendering , 2015, Comput. Graph. Forum.

[41]  Frédo Durand,et al.  Fast 4D Sheared Filtering for Interactive Rendering of Distribution Effects , 2015, ACM Trans. Graph..

[42]  Colin Raffel,et al.  Lasagne: First release. , 2015 .

[43]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[44]  Wenzel Jakob,et al.  layerlab : A computational toolbox for layered materials by , 2015 .

[45]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[46]  Thomas Brox,et al.  Learning to generate chairs with convolutional neural networks , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[47]  Viorica Patraucean,et al.  Spatio-temporal video autoencoder with differentiable memory , 2015, ArXiv.

[48]  Thomas Brox,et al.  FlowNet: Learning Optical Flow with Convolutional Networks , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[49]  Luca Fascione,et al.  The path tracing revolution in the movie industry , 2015, SIGGRAPH Courses.

[50]  Jian Sun,et al.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[51]  Kenny Mitchell,et al.  Adaptive rendering with linear predictions , 2015, ACM Trans. Graph..

[52]  Olga Sorkine-Hornung,et al.  Path‐space Motion Estimation and Decomposition for Robust Animation Filtering , 2015, Comput. Graph. Forum.

[53]  Daniel Rueckert,et al.  Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[54]  Jiri Matas,et al.  Systematic evaluation of CNN advances on the ImageNet , 2016, ArXiv.

[55]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[56]  Song Han,et al.  EIE: Efficient Inference Engine on Compressed Deep Neural Network , 2016, 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA).

[57]  Yu-Bin Yang,et al.  Image Restoration Using Convolutional Auto-encoders with Symmetric Skip Connections , 2016, ArXiv.

[58]  John Salvatier,et al.  Theano: A Python framework for fast computation of mathematical expressions , 2016, ArXiv.

[59]  Steven McDonagh,et al.  Adaptive polynomial rendering , 2016, ACM Trans. Graph..

[60]  John Flynn,et al.  Deep Stereo: Learning to Predict New Views from the World's Imagery , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[61]  Alexei A. Efros,et al.  Context Encoders: Feature Learning by Inpainting , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[62]  Gregory Shakhnarovich,et al.  Learning Representations for Automatic Colorization , 2016, ECCV.

[63]  Song Han,et al.  Deep compression and EIE: Efficient inference engine on compressed deep neural network , 2016, 2016 IEEE Hot Chips 28 Symposium (HCS).

[64]  Kenny Mitchell,et al.  Nonlinearly Weighted First‐order Regression for Denoising Monte Carlo Renderings , 2016, Comput. Graph. Forum.

[65]  Hiroshi Ishikawa,et al.  Let there be color! , 2016, ACM Trans. Graph..

[66]  Tomas Akenine-Möller,et al.  Texture space caching and reconstruction for ray tracing , 2016, ACM Trans. Graph..

[67]  Christopher Kulla,et al.  Physically based shading in theory and practice , 2014, SIGGRAPH '14.

[68]  Jan Kautz,et al.  Loss Functions for Image Restoration With Neural Networks , 2017, IEEE Transactions on Computational Imaging.

[69]  Mark Meyer,et al.  Kernel-predicting convolutional networks for denoising Monte Carlo renderings , 2017, ACM Trans. Graph..

[70]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[71]  Brian McWilliams,et al.  Kernel-Predicting Convolutional Networks for Denoising , 2017 .