Light Source Estimation with Analytical Path-tracing

We present a novel algorithm for light source estimation in scenes reconstructed with a RGB-D camera based on an analytically-derived formulation of path-tracing. Our algorithm traces the reconstructed scene with a custom path-tracer and computes the analytical derivatives of the light transport equation from principles in optics. These derivatives are then used to perform gradient descent, minimizing the photometric error between one or more captured reference images and renders of our current lighting estimation using an environment map parameterization for light sources. We show that our approach of modeling all light sources as points at infinity approximates lights located near the scene with surprising accuracy. Due to the analytical formulation of derivatives, optimization to the solution is considerably accelerated. We verify our algorithm using both real and synthetic data.

[1]  Jean-François Lalonde,et al.  Lighting Estimation in Outdoor Image Collections , 2014, 2014 2nd International Conference on 3D Vision.

[2]  Daniel Kurz,et al.  Real-time illumination estimation from faces for coherent rendering , 2014, 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[3]  Atsuto Maki,et al.  Self shadows and cast shadows in estimating illumination distribution , 2007 .

[4]  Vladlen Koltun,et al.  A Simple Model for Intrinsic Image Decomposition with Depth Cues , 2013, 2013 IEEE International Conference on Computer Vision.

[5]  Greg Humphreys,et al.  Physically Based Rendering, Second Edition: From Theory To Implementation , 2010 .

[6]  Andrew I. Comport,et al.  3D High Dynamic Range dense visual SLAM and its application to real-time object re-lighting , 2013, 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[7]  Shree K. Nayar,et al.  Radiometric self calibration , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[8]  Adrien Bousseau,et al.  Multiview Intrinsic Images of Outdoors Scenes with an Application to Relighting , 2015, ACM Trans. Graph..

[9]  Peter I. Corke,et al.  Dealing with shadows: Capturing intrinsic scene appearance for image-based outdoor localisation , 2013, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[10]  Daniel Cremers,et al.  A Super-Resolution Framework for High-Accuracy Multiview Reconstruction , 2013, International Journal of Computer Vision.

[11]  Olaf Kähler,et al.  Very High Frame Rate Volumetric Integration of Depth Images on Mobile Devices , 2015, IEEE Transactions on Visualization and Computer Graphics.

[12]  Marc Pollefeys,et al.  Robust Radiometric Calibration and Vignetting Correction , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  Thorsten Grosch,et al.  Interactive near-field illumination for photorealistic augmented reality on mobile devices , 2014, 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[14]  Bernard Ghanem,et al.  Intrinsic Scene Decomposition from RGB-D Images , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[15]  Stefan Leutenegger,et al.  ElasticFusion: Real-time dense SLAM and light source estimation , 2016, Int. J. Robotics Res..

[16]  J. B. Rosen The Gradient Projection Method for Nonlinear Programming. Part I. Linear Constraints , 1960 .

[17]  Wei Zhou,et al.  Estimation of Illuminant Direction and Intensity of Multiple Light Sources , 2002, ECCV.

[18]  Andrew J. Davison,et al.  Real-time surface light-field capture for augmentation of planar specular surfaces , 2012, 2012 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[19]  Katsushi Ikeuchi,et al.  Multiple light sources and reflectance property estimation based on a mixture of spherical distributions , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[20]  Vladlen Koltun,et al.  Color map optimization for 3D reconstruction with consumer depth cameras , 2014, ACM Trans. Graph..

[21]  Robert B. Fisher,et al.  Point Light Source Estimation based on Scenes Recorded by a RGB-D camera , 2013, BMVC.

[22]  Tito Homem-de-Mello,et al.  Monte Carlo sampling-based methods for stochastic optimization , 2014 .