Deep Light Source Estimation for Mixed Reality

Mixed reality is the union of virtual and real elements in a single scene. In this composition, of real and virtual elements, perceptual discrepancies in the illumination of objects may occur. We call these discrepancies the illumination mismatch problem. Recovering the lighting information from a real scene is a difficult task. Usually, such task requires prior knowledge of the scene, such as the scene geometry and special measuring equipment. We present a deep learning based technique that estimates point light source position from a single color image. The estimated light source position is used to create a composite image containing both the real and virtual environments. The proposed technique allows the final composite image to have consistent illumination between the real and virtual worlds, effectively reducing the effects of the illumination mismatch in Mixed Reality applications.

[1]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[2]  Alexei A. Efros,et al.  What Do the Sun and the Sky Tell Us About the Camera? , 2010, International Journal of Computer Vision.

[3]  Diego Gutierrez,et al.  Screen-Space Subsurface Scattering , 2010, GPU Pro.

[4]  Ling Shao,et al.  Enhanced Computer Vision With Microsoft Kinect Sensor: A Review , 2013, IEEE Transactions on Cybernetics.

[5]  Konstantinos Kamnitsas,et al.  Efficient multi‐scale 3D CNN with fully connected CRF for accurate brain lesion segmentation , 2016, Medical Image Anal..

[6]  Fei-Fei Li,et al.  ImageNet: A large-scale hierarchical image database , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[7]  Paul Graham,et al.  A single-shot light probe , 2012, SIGGRAPH '12.

[8]  Zhuowen Tu,et al.  Aggregated Residual Transformations for Deep Neural Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Paul E. Debevec.  Image-Based Lighting , 2002, IEEE Computer Graphics and Applications.

[10]  Veronica Teichrieb,et al.  RPR-SORS: Real-time photorealistic rendering of synthetic objects into real scenes , 2012, Comput. Graph..

[11]  Alexei A. Efros,et al.  Estimating the Natural Illumination Conditions from a Single Outdoor Image , 2012, International Journal of Computer Vision.

[12]  Pietro Zanuttigh,et al.  Hand gesture recognition with leap motion and kinect devices , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[13]  Vincent Lepetit,et al.  Learning Lightprobes for Mixed Reality Illumination , 2017, 2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[14]  Alexander Wilkie,et al.  An analytic model for full spectral sky-dome radiance , 2012, ACM Trans. Graph..

[15]  Sebastian Thrun,et al.  Dermatologist-level classification of skin cancer with deep neural networks , 2017, Nature.

[16]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Hristo Bojinov,et al.  Object Detection Using Deep CNNs Trained on Synthetic Images , 2017, ArXiv.

[18]  Pietro Perona,et al.  Microsoft COCO: Common Objects in Context , 2014, ECCV.

[19]  Robert B. Fisher,et al.  Interactive light source position estimation for augmented reality with an RGB‐D camera , 2017, Comput. Animat. Virtual Worlds.

[20]  Éric Marchand,et al.  Reflectance and Illumination Estimation for Realistic Augmentations of Real Scenes , 2016, 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct).

[21]  Yannick Hold-Geoffroy,et al.  Deep Outdoor Illumination Estimation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Álvaro González.  Measurement of Areas on a Sphere Using Fibonacci and Latitude–Longitude Lattices , 2009, 0912.4540.

[23]  Wei Liang,et al.  3D head pose estimation with convolutional neural network trained on synthetic images , 2016, 2016 IEEE International Conference on Image Processing (ICIP).

[24]  Martin Knecht,et al.  Reciprocal shading for mixed reality , 2012, Comput. Graph..

[25]  Kenny Mitchell,et al.  The shading probe: fast appearance acquisition for mobile AR , 2013, SA '13.

[26]  Luís Paulo Santos,et al.  Spherical Fibonacci Point Sets for Illumination Integrals , 2013, Comput. Graph. Forum.

[27]  Zhengyou Zhang,et al.  Microsoft Kinect Sensor and Its Effect , 2012, IEEE Multim..