Robust Point Light Source Estimation Using Differentiable Rendering

Illumination estimation is often used in mixed reality to re-render a scene from another point of view, to change the color/texture of an object, or to insert a virtual object consistently lit into a real video or photograph. Specifically, the estimation of a point light source is required for the shadows cast by the inserted object to be consistent with the real scene. We tackle the problem of illumination retrieval given an RGBD image of the scene as an inverse problem: we aim to find the illumination that minimizes the photometric error between the rendered image and the observation. In particular we propose a novel differentiable renderer based on the Blinn-Phong model with cast shadows. We compare our differentiable renderer to state-of-the-art methods and demonstrate its robustness to an incorrect reflectance estimation.

[1]  Philippe Robert,et al.  [POSTER] Illumination Estimation Using Cast Shadows for Realistic Augmented Reality Applications , 2017, 2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct).

[2]  Mario Fritz,et al.  Deep Reflectance Maps , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Alain Trémeau,et al.  Lighting Estimation in Indoor Environments from Low-Quality Images , 2012, ECCV Workshops.

[4]  Patrick Pérez,et al.  MoFA: Model-Based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[5]  James F. Blinn,et al.  Models of light reflection for computer synthesized pictures , 1977, SIGGRAPH.

[6]  Dieter Schmalstieg,et al.  Efficient and robust radiance transfer for probeless photorealistic augmented reality , 2014, 2014 IEEE Virtual Reality (VR).

[7]  Robert B. Fisher,et al.  Point Light Source Estimation based on Scenes Recorded by a RGB-D camera , 2013, BMVC.

[8]  Mario Fritz,et al.  Reflectance and Natural Illumination from Single-Material Specular Objects Using Deep Learning , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[9]  Carlos D. Castillo,et al.  SfSNet: Learning Shape, Reflectance and Illuminance of Faces 'in the Wild' , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[10]  Michael J. Black,et al.  OpenDR: An Approximate Differentiable Renderer , 2014, ECCV.

[11]  Tobias Ritschel,et al.  Joint Material and Illumination Estimation from Photo Sets in the Wild , 2017, 2018 International Conference on 3D Vision (3DV).

[12]  Ersin Yumer,et al.  Learning to predict indoor illumination from a single image , 2017, ACM Trans. Graph..

[13]  David A. Forsyth,et al.  Rendering synthetic objects into legacy photographs , 2011, ACM Trans. Graph..

[14]  Dieter Schmalstieg,et al.  Image-space illumination for augmented reality in dynamic environments , 2015, 2015 IEEE Virtual Reality (VR).

[15]  Erik Reinhard,et al.  Multiple Light Source Estimation in a Single Image , 2013, Comput. Graph. Forum.

[16]  Céline Loscos,et al.  Classification of Illumination Methods for Mixed Reality , 2006, Comput. Graph. Forum.

[17]  Katsushi Ikeuchi,et al.  Determining Reflectance Parameters and Illumination Distribution from a Sparse Set of Images for View-dependent Image Synthesis , 2001, ICCV.

[18]  M. Zollhöfer,et al.  Self-Supervised Multi-level Face Model Learning for Monocular Reconstruction at Over 250 Hz , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[19]  Yang Liu,et al.  Point Light Source Position Estimation From RGB-D Images by Learning Surface Attributes , 2017, IEEE Transactions on Image Processing.

[20]  Takahiro Okabe,et al.  Spherical harmonics vs. Haar wavelets: basis for recovering illumination from cast shadows , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[21]  Vincent Lepetit,et al.  Learning Lightprobes for Mixed Reality Illumination , 2017, 2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[22]  Éric Marchand,et al.  Reflectance and Illumination Estimation for Realistic Augmentations of Real Scenes , 2016, 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct).

[23]  Jiajun Wu,et al.  Self-Supervised Intrinsic Image Decomposition , 2017, NIPS.

[24]  Luc Van Gool,et al.  DeLight-Net: Decomposing Reflectance Maps into Specular Materials and Natural Illumination , 2016, ArXiv.

[25]  Bui Tuong Phong.  Illumination for computer generated pictures , 1975, Commun. ACM.

[26]  Kiyoshi Kiyokawa,et al.  Parallel lighting and reflectance estimation based on inverse rendering , 2013, 2013 23rd International Conference on Artificial Reality and Telexistence (ICAT).

[27]  Yannick Hold-Geoffroy,et al.  Deep Outdoor Illumination Estimation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Philippe Robert,et al.  Photometric Registration using Specular Reflections and Application to Augmented Reality , 2018 .

[29]  Dieter Schmalstieg,et al.  Real-time photometric registration from arbitrary geometry , 2012, 2012 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[30]  Katsushi Ikeuchi,et al.  Illumination Distribution from Brightness in Shadows: Adaptive Estimation of Illumination Distribution with Unknown Reflectance Properties in Shadow Regions , 1999, ICCV.

[31]  Katsushi Ikeuchi,et al.  Light source position and reflectance estimation from a single view without the distant illumination assumption , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.