Face relighting with radiance environment maps

A radiance environment map pre-integrates a constant surface reflectance with the lighting environment. It has been used to generate photo-realistic rendering at interactive speed. However, one of its limitations is that each radiance environment map can only render the object, which has the same surface reflectance as what it integrates. We present a ratio-image based technique to use a radiance environment map to render diffuse objects with different surface reflectance properties. This method has the advantage that it does not require the separation of illumination from reflectance, and it is simple to implement and runs at interactive speed. In order to use this technique for human face relighting, we have developed a technique that uses spherical harmonics to approximate the radiance environment map for any given image of a face. Thus we are able to relight face images when the lighting environment rotates. Another benefit of the radiance environment map is that we can interactively modify lighting by changing the coefficients of the spherical harmonics basis. Finally we can modify the lighting condition of one person's face so that it matches the new lighting condition of a different person's face image assuming the two faces have similar skin albedos.

[1]  Li Zhang,et al.  Single view modeling of free-form scenes , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[2]  George Drettakis,et al.  Interactive Virtual Relighting of Real Scenes , 2000, IEEE Trans. Vis. Comput. Graph..

[3]  Pat Hanrahan,et al.  An efficient representation for irradiance environment maps , 2001, SIGGRAPH.

[4]  E. Land,et al.  Lightness and retinex theory. , 1971, Journal of the Optical Society of America.

[5]  Zicheng Liu,et al.  Expressive expression mapping with ratio images , 2001, SIGGRAPH.

[6]  Steve Marschner,et al.  Inverse Lighting for Photography , 1997, CIC.

[7]  Paul E. Debevec,et al.  Rendering synthetic objects into real scenes: bridging traditional and image-based graphics with global illumination and high dynamic range photography , 1998, SIGGRAPH '08.

[8]  B. Ripley,et al.  Robust Statistics , 2018, Encyclopedia of Mathematical Geosciences.

[9]  David J. Kriegman,et al.  Illumination-based image synthesis: creating novel images of human faces under differing pose and lighting , 1999, Proceedings IEEE Workshop on Multi-View Modeling and Analysis of Visual Scenes (MVIEW'99).

[10]  Marc Olano,et al.  Reflection space image based rendering , 1999, SIGGRAPH.

[11]  Katsushi Ikeuchi,et al.  Acquiring a Radiance Distribution to Superimpose Virtual Objects onto Real Scene , 2001, MVA.

[12]  David Salesin,et al.  Image Analogies , 2001, SIGGRAPH.

[13]  C. R. Hoffman,et al.  Illumination and Reflection Maps : Simulated Objects in Simulated and Real Environments Gene , 1984 .

[14]  Steve Marschner,et al.  Modeling and Rendering for Realistic Facial Animation , 2000, Rendering Techniques.

[15]  Paul Debevec,et al.  Inverse global illumination: Recovering re?ectance models of real scenes from photographs , 1998 .

[16]  Ronen Basri,et al.  Lambertian Reflectance and Linear Subspaces , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[17]  Thomas Vetter,et al.  A morphable model for the synthesis of 3D faces , 1999, SIGGRAPH.

[18]  Amnon Shashua,et al.  The quotient image: Class based recognition and synthesis under varying illumination conditions , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[19]  Ravi Ramamoorthi,et al.  Analytic PCA Construction for Theoretical Analysis of Lighting Variability in Images of a Lambertian Object , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[20]  Steve Marschner,et al.  Image-Based BRDF Measurement Including Human Skin , 1999, Rendering Techniques.

[21]  J. Linnett,et al.  Quantum mechanics , 1975, Nature.

[22]  Katsushi Ikeuchi,et al.  Determining Reflectance Parameters and Illumination Distribution from a Sparse Set of Images for View-dependent Image Synthesis , 2001, ICCV.

[23]  Arne Stoschek,et al.  Image-based re-rendering of faces for continuous pose and illumination directions , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[24]  S. Marschner,et al.  Inverse Rendering for Computer Graphics , 1998 .

[25]  Ned Greene,et al.  Environment Mapping and Other Applications of World Projections , 1986, IEEE Computer Graphics and Applications.

[26]  Pat Hanrahan,et al.  A signal-processing framework for inverse rendering , 2001, SIGGRAPH.

[27]  Paul E. Debevec,et al.  Acquiring the reflectance field of a human face , 2000, SIGGRAPH.