DeepLight: light source estimation for augmented reality using deep learning

This paper presents a novel method for illumination estimation from RGB-D images. The main focus of the proposed method is to enhance visual coherence in augmented reality applications by providing accurate and temporally coherent estimates of real illumination. For this purpose, we designed and trained a deep neural network which calculates a dominant light direction from a single RGB-D image. Additionally, we propose a novel method for real-time outlier detection to achieve temporally coherent estimates. Our method for light source estimation in augmented reality was evaluated on the set of real scenes. Our results demonstrate that the neural network can successfully estimate light sources even in scenes which were not seen by the network during training. Moreover, we compared our results with illumination estimates calculated by the state-of-the-art method for illumination estimation. Finally, we demonstrate the applicability of our method on numerous augmented reality scenes.

[1]  Pieter Peers,et al.  Appearance-from-motion , 2014, ACM Trans. Graph..

[2]  Erik Reinhard,et al.  Multiple Light Source Estimation in a Single Image , 2013, Comput. Graph. Forum.

[3]  Katsushi Ikeuchi,et al.  Illumination from Shadows , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[4]  Yannick Hold-Geoffroy,et al.  Deep Outdoor Illumination Estimation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Pat Hanrahan,et al.  A signal-processing framework for inverse rendering , 2001, SIGGRAPH.

[6]  Stephen Lin,et al.  Shading-Based Shape Refinement of RGB-D Images , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[7]  Daniel Kurz,et al.  Real-time illumination estimation from faces for coherent rendering , 2014, ISMAR.

[8]  Dieter Schmalstieg,et al.  Image-space illumination for augmented reality in dynamic environments , 2015, 2015 IEEE Virtual Reality (VR).

[9]  Koch,et al.  Markerless Augmented Reality with Light Source Estimation for Direct Illumination , 2006 .

[10]  Roberto Cipolla,et al.  A Practical Method for Estimation of Point Light-Sources , 2001, BMVC.

[11]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Vincent Lepetit,et al.  Learning Lightprobes for Mixed Reality Illumination , 2017, 2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[13]  Stefan Leutenegger,et al.  ElasticFusion: Real-time dense SLAM and light source estimation , 2016, Int. J. Robotics Res..

[14]  Dieter Schmalstieg,et al.  Instant Mixed Reality Lighting from Casual Scanning , 2016, 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[15]  Dieter Schmalstieg,et al.  Real-time photometric registration from arbitrary geometry , 2012, 2012 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[16]  Wei Zhou,et al.  Estimation of Illuminant Direction and Intensity of Multiple Light Sources , 2002, ECCV.

[17]  Geoffrey E. Hinton,et al.  Rectified Linear Units Improve Restricted Boltzmann Machines , 2010, ICML.

[18]  Hannes Kaufmann,et al.  High-Quality Consistent Illumination in Mobile Augmented Reality by Radiance Convolution on the GPU , 2015, ISVC.

[19]  David A. Elizondo,et al.  Light source detection for digital images in noisy scenes: a neural network approach , 2016, Neural Computing and Applications.

[20]  Li Li,et al.  Photorealistic rendering for augmented reality using environment illumination , 2003, The Second IEEE and ACM International Symposium on Mixed and Augmented Reality, 2003. Proceedings..

[21]  Éric Marchand,et al.  Reflectance and Illumination Estimation for Realistic Augmentations of Real Scenes , 2016, 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct).

[22]  Paul Debevec.  Rendering synthetic objects into real scenes: bridging traditional and image-based graphics with global illumination and high dynamic range photography , 2008, SIGGRAPH Classes.

[23]  Martin Knecht,et al.  Differential Instant Radiosity for mixed reality , 2010, 2010 IEEE International Symposium on Mixed and Augmented Reality.

[24]  Esteban Walter Gonzalez Clua,et al.  Deep Light Source Estimation for Mixed Reality , 2018, VISIGRAPP.

[25]  Robert B. Fisher,et al.  Interactive light source position estimation for augmented reality with an RGB‐D camera , 2017, Comput. Animat. Virtual Worlds.

[26]  Robert B. Fisher,et al.  Point Light Source Estimation based on Scenes Recorded by a RGB-D camera , 2013, BMVC.

[27]  Gabe Sibley,et al.  Light Source Estimation with Analytical Path-tracing , 2017, ArXiv.

[28]  Kalyan Sunkavalli,et al.  Automatic Scene Inference for 3D Object Compositing , 2014, ACM Trans. Graph..

[29]  Michael Haller,et al.  Image Based Shadowing in Real-Time Augmented Reality , 2006, Int. J. Virtual Real..

[30]  Peter Kán,et al.  High-Quality Real-Time Global Illumination in Augmented Reality , 2014 .

[31]  Ersin Yumer,et al.  Learning to predict indoor illumination from a single image , 2017, ACM Trans. Graph..

[32]  Mike Brookes,et al.  Precise real-time outlier removal from motion vector fields for 3D reconstruction , 2003, Proceedings 2003 International Conference on Image Processing (Cat. No.03CH37429).

[33]  Thorsten Grosch,et al.  Interactive near-field illumination for photorealistic augmented reality on mobile devices , 2014, 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[34]  Dieter Schmalstieg,et al.  ARToolKitPlus for Pose Trackin on Mobile Devices , 2007 .