Learning Lightprobes for Mixed Reality Illumination

This paper presents the first photometric registration pipeline for Mixed Reality based on high quality illumination estimation using convolutional neural networks (CNNs). For easy adaptation and deployment of the system, we train the CNNs using purely synthetic images and apply them to real image data. To keep the pipeline accurate and efficient, we propose to fuse the light estimation results from multiple CNN instances and show an approach for caching estimates over time. For optimal performance, we furthermore explore multiple strategies for the CNN training. Experimental results show that the proposed method yields highly accurate estimates for photo-realistic augmentations.

[1]  David W. Murray,et al.  Simulating Low-Cost Cameras for Augmented Reality Compositing , 2010, IEEE Transactions on Visualization and Computer Graphics.

[2]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[3]  Dieter Schmalstieg,et al.  Real-time photometric registration from arbitrary geometry , 2012, 2012 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[4]  David W. Murray,et al.  Compositing for small cameras , 2008, 2008 7th IEEE/ACM International Symposium on Mixed and Augmented Reality.

[5]  Dieter Schmalstieg,et al.  Instant Mixed Reality Lighting from Casual Scanning , 2016, 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[6]  Yoshua Bengio,et al.  Deep Sparse Rectifier Neural Networks , 2011, AISTATS.

[7]  David E. Breen,et al.  Interactive Occlusion and Automatic Object Placement for Augmented Reality , 1996, Comput. Graph. Forum.

[8]  Andrew J. Davison,et al.  Real-time surface light-field capture for augmentation of planar specular surfaces , 2012, 2012 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[9]  John Salvatier,et al.  Theano: A Python framework for fast computation of mathematical expressions , 2016, ArXiv.

[10]  David W. Murray,et al.  Real-time localization and mapping with wearable active vision , 2003, The Second IEEE and ACM International Symposium on Mixed and Augmented Reality, 2003. Proceedings..

[11]  Paul E. Debevec,et al.  Rendering synthetic objects into real scenes: bridging traditional and image-based graphics with global illumination and high dynamic range photography , 1998, SIGGRAPH '08.

[12]  Taehyun Rhee,et al.  MR360: Mixed Reality Rendering for 360° Panoramic Videos , 2017, IEEE Transactions on Visualization and Computer Graphics.

[13]  Dieter Schmalstieg,et al.  Efficient and robust radiance transfer for probeless photorealistic augmented reality , 2014, 2014 IEEE Virtual Reality (VR).

[14]  David W. Murray,et al.  Video-rate localization in multiple maps for wearable augmented reality , 2008, 2008 12th IEEE International Symposium on Wearable Computers.

[15]  G. Klein,et al.  Parallel Tracking and Mapping for Small AR Workspaces , 2007, 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality.

[16]  Pascal Fua,et al.  Using Specularities to Recover Multiple Light Sources in the Presence of Texture , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[17]  Daniel Kurz,et al.  Real-time illumination estimation from faces for coherent rendering , 2014, 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[18]  Tobias Höllerer,et al.  Augmented reality: principles and practice , 2016, SIGGRAPH Courses.

[19]  Gerhard Reitmayr,et al.  Coherent image-based rendering of real-world objects , 2011, SI3D.

[20]  Vincent Lepetit,et al.  An all-in-one solution to geometric and photometric calibration , 2006, 2006 IEEE/ACM International Symposium on Mixed and Augmented Reality.

[21]  Martin Knecht,et al.  Differential Instant Radiosity for mixed reality , 2010, 2010 IEEE International Symposium on Mixed and Augmented Reality.

[22]  Robin Green,et al.  Spherical Harmonic Lighting: The Gritty Details , 2003 .

[23]  Dieter Schmalstieg,et al.  Image-space illumination for augmented reality in dynamic environments , 2015, 2015 IEEE Virtual Reality (VR).