Lightweight face relighting

We present a method to relight human faces in real time, using consumer-grade graphics cards even with limited 3D capabilities. We show how to render faces using a combination of a simple, hardware-accelerated parametric model simulating skin shading and a detail texture map, and provide robust procedures to estimate all the necessary parameters for a given face. Our model strikes a balance between the difficulty of realistic face rendering (given the very specific reflectance properties of skin) and the goal of real-time rendering with limited hardware capabilities. This is accomplished by automatically generating an optimal set of parameters for a simple rendering model. We offer a discussion of the issues in face rendering to discern the pros and cons of various rendering models and to generalize our approach to most of the current hardware constraints. We provide results demonstrating the usability of our approach and the improvements we introduce both in the performance and in the visual quality of the resulting faces.

[1]  G. Wyszecki,et al.  Color Science Concepts and Methods , 1982 .

[2]  Guillermo Sapiro,et al.  Image inpainting , 2000, SIGGRAPH.

[3]  Gabriel Taubin,et al.  Acquiring Input for Rendering at Appropriate Levels of Detail: Digitizing a Tietà , 1998, Rendering Techniques.

[4]  Simon Premoze,et al.  Analytic light transport approximations for volumetric materials , 2002, 10th Pacific Conference on Computer Graphics and Applications, 2002. Proceedings..

[5]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[6]  Jan Kautz,et al.  Precomputed radiance transfer for real-time rendering in dynamic, low-frequency lighting environments , 2002 .

[7]  Long Quan,et al.  Quasi-Dense Reconstruction from Image Sequence , 2002, ECCV.

[8]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[9]  David J. Kriegman,et al.  Illumination-based image synthesis: creating novel images of human faces under differing pose and lighting , 1999, Proceedings IEEE Workshop on Multi-View Modeling and Analysis of Visual Scenes (MVIEW'99).

[10]  Steven A. Shafer,et al.  Using color to separate reflection components , 1985 .

[11]  Andreas Kolb,et al.  Homomorphic factorization of BRDF-based lighting computation , 2002, ACM Trans. Graph..

[12]  Y. J. Tejwani,et al.  Robot vision , 1989, IEEE International Symposium on Circuits and Systems,.

[13]  Lance Williams,et al.  Casting curved shadows on curved surfaces , 1978, SIGGRAPH.

[14]  Matthew Turk,et al.  A Morphable Model For The Synthesis Of 3D Faces , 1999, SIGGRAPH.

[15]  Hans-Peter Seidel,et al.  Towards interactive bump mapping with anisotropic shift-variant BRDFs , 2000, Workshop on Graphics Hardware.

[16]  John M. Snyder,et al.  Parameterized Animation Compression , 2000, Rendering Techniques.

[17]  Henrik Wann Jensen,et al.  A rapid hierarchical rendering technique for translucent materials , 2005, ACM Trans. Graph..

[18]  Andrew Gardner,et al.  A lighting reproduction approach to live-action compositing , 2002, SIGGRAPH.

[19]  Hans-Peter Seidel,et al.  Efficient Cloth Modeling and Rendering , 2001, Rendering Techniques.

[20]  Anselmo Lastra,et al.  A generalized surface appearance representation for computer graphics , 2002 .

[21]  Hans-Peter Seidel,et al.  Walkthroughs with Corrective Texturing , 2000, Rendering Techniques.

[22]  Katsushi Ikeuchi,et al.  Object shape and reflectance modeling from observation , 1997, SIGGRAPH.

[23]  Zicheng Liu,et al.  Model-based bundle adjustment with application to face modeling , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[24]  Zicheng Liu,et al.  Expressive expression mapping with ratio images , 2001, SIGGRAPH.

[25]  O. Faugeras,et al.  The Geometry of Multiple Images , 1999 .

[26]  Anselmo Lastra,et al.  Efficient rendering of spatial bi-directional reflectance distribution functions , 2002, HWWS '02.

[27]  Olivier D. Faugeras,et al.  The geometry of multiple images - the laws that govern the formation of multiple images of a scene and some of their applications , 2001 .

[28]  Pierre Poulin,et al.  Interactive Virtual Relighting and Remodeling of Real Scenes , 1999, Rendering Techniques.

[29]  Kosuke Sato,et al.  Determining Reflectance Properties of an Object Using Range and Brightness Images , 1991, IEEE Trans. Pattern Anal. Mach. Intell..

[30]  André Gagalowicz,et al.  Image-based rendering of diffuse, specular and glossy surfaces from a single image , 2001, SIGGRAPH.

[31]  Hans-Peter Seidel,et al.  Image-Based Reconstruction of Spatially Varying Materials , 2001 .

[32]  Zicheng Liu,et al.  Face relighting with radiance environment maps , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[33]  Steve Marschner,et al.  Inverse Lighting for Photography , 1997, CIC.

[34]  Steve Marschner,et al.  Modeling and Rendering for Realistic Facial Animation , 2000, Rendering Techniques.

[35]  Hans-Peter Seidel,et al.  Texturing Faces , 2002, Graphics Interface.

[36]  Katsushi Ikeuchi,et al.  Determining Reflectance Parameters and Illumination Distribution from a Sparse Set of Images for View-dependent Image Synthesis , 2001, ICCV.

[37]  Long Quan,et al.  Surface reconstruction by integrating 3D and 2D data of multiple views , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[38]  Gunther Wyszecki,et al.  Color Science: Concepts and Methods, Quantitative Data and Formulae, 2nd Edition , 2000 .

[39]  Steve Marschner,et al.  Image-Based BRDF Measurement Including Human Skin , 1999, Rendering Techniques.

[40]  Steve Marschner,et al.  A practical model for subsurface light transport , 2001, SIGGRAPH.

[41]  Bernice E. Rogowitz,et al.  Perceptual issues in substituting texture for geometry , 2000, Electronic Imaging.

[42]  Holly E. Rushmeier,et al.  Computing consistent normals and colors from photometric data , 1999, Second International Conference on 3-D Digital Imaging and Modeling (Cat. No.PR00062).

[43]  J. Cohen,et al.  Color Science: Concepts and Methods, Quantitative Data and Formulas , 1968 .

[44]  Zicheng Liu,et al.  Robust and Rapid Generation of Animated Faces from Video Images: A Model-Based Modeling Approach , 2004, International Journal of Computer Vision.

[45]  Donald P. Greenberg,et al.  Non-linear approximation of reflectance functions , 1997, SIGGRAPH.

[46]  Pat Hanrahan,et al.  Frequency space environment map rendering , 2002, SIGGRAPH.

[47]  K. Torrance,et al.  Theory for off-specular reflection from roughened surfaces , 1967 .

[48]  Paul E. Debevec,et al.  Acquiring the reflectance field of a human face , 2000, SIGGRAPH.