Real-time illumination estimation from faces for coherent rendering

We present a method for estimating the real-world lighting conditions within a scene in real-time. The estimation is based on the visual appearance of a human face in the real scene captured in a single image of a monocular camera. In hardware setups featuring a user-facing camera, an image of the user's face can be acquired at any time. The limited range in variations between different human faces makes it possible to analyze their appearance offline, and to apply the results to new faces. Our approach uses radiance transfer functions - learned offline from a dataset of images of faces under different known illuminations - for particular points on the human face. Based on these functions, we recover the most plausible real-world lighting conditions for measured reflections in a face, represented by a function depending on incident light angle using Spherical Harmonics. The pose of the camera relative to the face is determined by means of optical tracking, and virtual 3D content is rendered and overlaid onto the real scene with a fixed spatial relationship to the face. By applying the estimated lighting conditions to the rendering of the virtual content, the augmented scene is shaded coherently with regard to the real and virtual parts of the scene. We show with different examples under a variety of lighting conditions, that our approach provides plausible results, which considerably enhance the visual realism in real-time Augmented Reality applications.

[1]  Robin Green,et al.  Spherical Harmonic Lighting: The Gritty Details , 2003 .

[2]  Andrew I. Comport,et al.  3D High Dynamic Range dense visual SLAM and its application to real-time object re-lighting , 2013, 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[3]  Miika Aittala,et al.  Inverse lighting and photorealistic rendering for augmented reality , 2010, The Visual Computer.

[4]  Steve Marschner,et al.  Inverse Lighting for Photography , 1997, CIC.

[5]  Jon Y. Hardeberg,et al.  Realtime Estimation of Illumination Direction for Augmented Reality on Mobile Devices , 2012, Color Imaging Conference.

[6]  Hans-Peter Seidel,et al.  Reflectance from images: a model-based approach for human faces , 2005, IEEE Transactions on Visualization and Computer Graphics.

[7]  Shree K. Nayar,et al.  Eyes for relighting , 2004, SIGGRAPH 2004.

[8]  Harry Shum,et al.  Eurographics Symposium on Rendering (2004) All-frequency Precomputed Radiance Transfer for Glossy Objects , 2022 .

[9]  Andrew J. Davison,et al.  Real-time surface light-field capture for augmentation of planar specular surfaces , 2012, 2012 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[10]  Lei Zhang,et al.  Face synthesis and recognition from a single image under arbitrary unknown lighting using a spherical harmonic basis morphable model , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[11]  F. E. Nicodemus,et al.  Geometrical considerations and nomenclature for reflectance , 1977 .

[12]  Tomoyuki Nishita,et al.  A montage method: the overlaying of the computer generated images onto a background photograph , 1986, SIGGRAPH.

[13]  David J. Kriegman,et al.  From Few to Many: Illumination Cone Models for Face Recognition under Variable Lighting and Pose , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[14]  Martin Knecht,et al.  Reciprocal shading for mixed reality , 2012, Comput. Graph..

[15]  Thomas Vetter,et al.  A morphable model for the synthesis of 3D faces , 1999, SIGGRAPH.

[16]  Ravi Ramamoorthi,et al.  Analytic PCA Construction for Theoretical Analysis of Lighting Variability in Images of a Lambertian Object , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[17]  M. Landy,et al.  The Plenoptic Function and the Elements of Early Vision , 1991 .

[18]  Lei Zhang,et al.  Face recognition under variable lighting using harmonic image exemplars , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[19]  Wen Gao,et al.  Eigen-harmonics faces: face recognition under generic lighting , 2004, Sixth IEEE International Conference on Automatic Face and Gesture Recognition, 2004. Proceedings..

[20]  Katsushi Ikeuchi,et al.  Acquiring a Radiance Distribution to Superimpose Virtual Objects onto Real Scene , 2001, MVA.

[21]  Alexander Keller,et al.  Instant radiosity , 1997, SIGGRAPH.

[22]  James T. Kajiya,et al.  The rendering equation , 1998 .

[23]  A. Fournier,et al.  Common Illumination between Real and Computer Generated Scenes , 1992 .

[24]  Paul Debevec.  Rendering synthetic objects into real scenes: bridging traditional and image-based graphics with global illumination and high dynamic range photography , 2008, SIGGRAPH Classes.

[25]  Ravi Ramamoorthi,et al.  Modeling Illumination Variation with Spherical Harmonics , 2005 .

[26]  Russell A. Epstein,et al.  5/spl plusmn/2 eigenimages suffice: an empirical investigation of low-dimensional lighting models , 1995, Proceedings of the Workshop on Physics-Based Modeling in Computer Vision.

[27]  Ronen Basri,et al.  Lambertian Reflectance and Linear Subspaces , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[28]  Akira Kojima,et al.  The hand as a shading probe , 2013, SIGGRAPH '13.

[29]  Kenny Mitchell,et al.  The shading probe: fast appearance acquisition for mobile AR , 2013, SA '13.

[30]  Pat Hanrahan,et al.  A signal-processing framework for inverse rendering , 2001, SIGGRAPH.

[31]  Paul E. Debevec,et al.  Acquiring the reflectance field of a human face , 2000, SIGGRAPH.

[32]  David W. Murray,et al.  Simulating Low-Cost Cameras for Augmented Reality Compositing , 2010, IEEE Transactions on Visualization and Computer Graphics.

[33]  Naokazu Yokoya,et al.  Real-time estimation of light source environment for photorealistic augmented reality , 2004, Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004..

[34]  Dieter Schmalstieg,et al.  Real-time photometric registration from arbitrary geometry , 2012, 2012 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[35]  A. Kuijpers-Jagtman.  [Illuminating the face]. , 1993, Nederlands tijdschrift voor tandheelkunde.

[36]  David J. Kriegman,et al.  Acquiring linear subspaces for face recognition under variable lighting , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.