Efficient and Differentiable Shadow Computation for Inverse Problems

Differentiable rendering has received increasing interest for image-based inverse problems. It can benefit traditional optimization-based solutions to inverse problems, but also allows for self-supervision of learning-based approaches for which training data with ground truth annotation is hard to obtain. However, existing differentiable renderers either do not model visibility of the light sources from the different points in the scene, responsible for shadows in the images, or are very slow which makes it difficult to train deep architectures over thousands of iterations. To this end, we propose an accurate yet efficient approach for differentiable visibility and soft shadow computation. Our approach is based on the spherical harmonics approximations of the scene illumination and visibility, where the occluding surface is approximated with spheres. This allows for a significantly more efficient shadow computation compared to methods based on ray tracing. As our formulation is differentiable, it can be used to solve inverse problems such as texture, illumination, rigid pose, and geometric deformation recovery from images using analysis-by-synthesis optimization.

[1]  Pablo Garrido,et al.  High-Fidelity Monocular Face Reconstruction Based on an Unsupervised Model-Based Face Autoencoder , 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[2]  Michael J. Black,et al.  OpenDR: An Approximate Differentiable Renderer , 2014, ECCV.

[3]  Yannick Hold-Geoffroy,et al.  Deep Sky Modeling for Single Image Outdoor Lighting Estimation , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Wan-Yen Lo,et al.  Accelerating 3D deep learning with PyTorch3D , 2019, SIGGRAPH Asia 2020 Courses.

[5]  Samuli Laine,et al.  Ambient occlusion fields , 2005, I3D '05.

[6]  Kei Iwasaki,et al.  Precomputed Radiance Transfer for Dynamic Scenes Taking into Account Light Interreflection , 2007, Rendering Techniques.

[7]  Wenzel Jakob,et al.  Reparameterizing discontinuous integrands for differentiable rendering , 2019, ACM Trans. Graph..

[8]  Matthias Nießner,et al.  Inverse Path Tracing for Joint Material and Lighting Estimation , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  M. Pauly,et al.  Embedded deformation for shape manipulation , 2007, SIGGRAPH 2007.

[10]  Matthew Turk,et al.  A Morphable Model For The Synthesis Of 3D Faces , 1999, SIGGRAPH.

[11]  Justus Thies,et al.  Deferred Neural Rendering: Image Synthesis using Neural Textures , 2019 .

[12]  Kun Zhou,et al.  Real-time soft shadows in dynamic scenes using spherical harmonic exponentiation , 2006, ACM Trans. Graph..

[13]  Kazufumi Kaneda,et al.  A Quick Rendering Method Using Basis Functions for Interactive Lighting Design , 1995, Comput. Graph. Forum.

[14]  James T. Kajiya,et al.  The rendering equation , 1986, SIGGRAPH.

[15]  Kun Zhou,et al.  Variational sphere set approximation for solid objects , 2006, The Visual Computer.

[16]  Kun Zhou,et al.  Precomputed shadow fields for dynamic scenes , 2005, ACM Trans. Graph..

[17]  Tatsuya Harada,et al.  Neural 3D Mesh Renderer , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[18]  Kalyan Sunkavalli,et al.  Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF From a Single Image , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Robin Green,et al.  Spherical Harmonic Lighting: The Gritty Details , 2003 .

[20]  Michael Wimmer,et al.  Real‐Time Indirect Illumination and Soft Shadows in Dynamic Scenes Using Spherical Lights , 2008, Comput. Graph. Forum.

[21]  Tatsuya Harada,et al.  Learning View Priors for Single-View 3D Reconstruction , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Ravi Ramamoorthi,et al.  A differential theory of radiative transfer , 2019, ACM Trans. Graph..

[23]  M. Gross,et al.  Analysis of human faces using a measurement-based skin reflectance model , 2006, ACM Trans. Graph..

[24]  Christian Theobalt,et al.  DeepCap: Monocular Human Performance Capture Using Weak Supervision , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Peter-Pike J. Sloan,et al.  Local, deformable precomputed radiance transfer , 2005, ACM Trans. Graph..

[26]  Jaakko Lehtinen,et al.  Modular primitives for high-performance differentiable rendering , 2020, ACM Trans. Graph..

[27]  Jaakko Lehtinen,et al.  Differentiable Monte Carlo ray tracing through edge sampling , 2018, ACM Trans. Graph..

[28]  Hao Li,et al.  Soft Rasterizer: A Differentiable Renderer for Image-Based 3D Reasoning , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[29]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[30]  Christian Theobalt,et al.  LiveCap , 2018, ACM Trans. Graph..

[31]  Jaakko Lehtinen,et al.  Hemispherical Rasterization for Self-Shadowing of Dynamic Objects , 2004, Rendering Techniques.

[32]  Jan Kautz,et al.  Fast Arbitrary BRDF Shading for Low-Frequency Lighting Using Spherical Harmonics , 2002, Rendering Techniques.

[33]  Ersin Yumer,et al.  Learning to predict indoor illumination from a single image , 2017, ACM Trans. Graph..

[34]  Frédo Durand,et al.  Unbiased warped-area sampling for differentiable rendering , 2020, ACM Trans. Graph..

[35]  Pat Hanrahan,et al.  An efficient representation for irradiance environment maps , 2001, SIGGRAPH.

[36]  Shuang Zhao,et al.  Physics-based differentiable rendering: from theory to implementation , 2020, SIGGRAPH Courses.

[37]  Patrick Pérez,et al.  MoFA: Model-Based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[38]  Cheng Zhang,et al.  Path-space differentiable rendering , 2020, ACM Trans. Graph..

[39]  Hans-Peter Seidel,et al.  A Versatile Scene Model with Differentiable Visibility Applied to Generative Pose Estimation , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[40]  Jan Kautz,et al.  The State of the Art in Interactive Global Illumination , 2012, Comput. Graph. Forum.

[41]  Adrien Bousseau,et al.  Single-image SVBRDF capture with a rendering-aware deep network , 2018, ACM Trans. Graph..

[42]  Jan Kautz,et al.  Precomputed radiance transfer for real-time rendering in dynamic, low-frequency lighting environments , 2002 .

[43]  M. Pollefeys,et al.  Precomputed Radiance Transfer for Reflectance and Lighting Estimation , 2020, 2020 International Conference on 3D Vision (3DV).