Analysis of human faces using a measurement-based skin reflectance model

We have measured 3D face geometry, skin reflectance, and subsurface scattering using custom-built devices for 149 subjects of varying age, gender, and race. We developed a novel skin reflectance model whose parameters can be estimated from measurements. The model decomposes the large amount of measured skin data into a spatially-varying analytic BRDF, a diffuse albedo map, and diffuse subsurface scattering. Our model is intuitive, physically plausible, and -- since we do not use the original measured data -- easy to edit as well. High-quality renderings come close to reproducing real photographs. The analysis of the model parameters for our sample population reveals variations according to subject age, gender, skin type, and external factors (e.g., sweat, cold, or makeup). Using our statistics, a user can edit the overall appearance of a face (e.g., changing skin type and age) or change small-scale features using texture synthesis (e.g., adding moles and freckles). We are making the collected statistics publicly available to the research community for applications in face synthesis and analysis.

[1]  Mark Pauly,et al.  Point primitives for interactive modeling and processing of 3D-geometry , 2003 .

[2]  David J. Kriegman,et al.  Illumination-based image synthesis: creating novel images of human faces under differing pose and lighting , 1999, Proceedings IEEE Workshop on Multi-View Modeling and Analysis of Visual Scenes (MVIEW'99).

[3]  Anil K. Jain Fundamentals of Digital Image Processing , 2018, Control of Color Imaging Systems.

[4]  Guillermo Sapiro,et al.  Inpainting surface holes , 2003, Proceedings 2003 International Conference on Image Processing (Cat. No.03CH37429).

[5]  Stephen H. Westin,et al.  Image-based bidirectional reflectance distribution function measurement. , 2000, Applied optics.

[6]  Julie Dorsey,et al.  Rendering of Wet Materials , 1999, Rendering Techniques.

[7]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[8]  Ramesh Raskar,et al.  Image-based visual hulls , 2000, SIGGRAPH.

[9]  Long Quan,et al.  Lightweight face relighting , 2003, 11th Pacific Conference onComputer Graphics and Applications, 2003. Proceedings..

[10]  David J. Kriegman,et al.  Beyond Lambert: reconstructing specular surfaces using color , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[11]  Frédo Durand,et al.  Experimental analysis of BRDF models , 2005, EGSR '05.

[12]  Wojciech Matusik,et al.  Opacity light fields: interactive rendering of surface light fields with view-dependent opacity , 2003, I3D '03.

[13]  Katsushi Ikeuchi,et al.  Appearance compression and synthesis based on 3D model for mixed reality , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[14]  Christian Sigg,et al.  Representation and rendering of implicit surfaces , 2006 .

[15]  Charles T. Loop,et al.  Smooth Subdivision Surfaces Based on Triangles , 1987 .

[16]  Andrew Gardner,et al.  Performance relighting and reflectance transformation with time-multiplexed illumination , 2005, SIGGRAPH 2005.

[17]  B. Funt,et al.  Diagonal versus affine transformations for color correction. , 2000, Journal of the Optical Society of America. A, Optics, image science, and vision.

[18]  T. Fitzpatrick The validity and practicality of sun-reactive skin types I through VI. , 1988, Archives of Dermatology.

[19]  F. E. Nicodemus,et al.  Geometrical considerations and nomenclature for reflectance , 1977 .

[20]  Hans-Peter Seidel,et al.  DISCO: acquisition of translucent objects , 2004, ACM Trans. Graph..

[21]  James F. Blinn,et al.  Models of light reflection for computer synthesized pictures , 1977, SIGGRAPH.

[22]  Elli Angelopoulou,et al.  Multispectral skin color modeling , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[23]  J. Sethian,et al.  Fronts propagating with curvature-dependent speed: algorithms based on Hamilton-Jacobi formulations , 1988 .

[24]  Kristin J. Dana,et al.  Skin Texture Modeling , 2005, International Journal of Computer Vision.

[25]  Tim Weyrich,et al.  Using a Panoramic Camera for 3D Head Tracking in an AR Environment , 2002 .

[26]  Steve Marschner,et al.  A practical model for subsurface light transport , 2001, SIGGRAPH.

[27]  Pedro V. Sander,et al.  Real-time skin rendering on graphics hardware , 2004, SIGGRAPH '04.

[28]  Katsushi Ikeuchi,et al.  Appearance Based Object Modeling using Texture Database: Acquisition Compression and Rendering , 2002, Rendering Techniques.

[29]  Ronald N. Perry,et al.  Simple and Efficient Traversal Methods for Quadtrees and Octrees , 2002, J. Graphics, GPU, & Game Tools.

[30]  Hans-Peter Seidel,et al.  Reflectance from images: a model-based approach for human faces , 2005, IEEE Transactions on Visualization and Computer Graphics.

[31]  Matthias Zwicker,et al.  3 Ideal Resampling 3 . 1 Sampling and Aliasing , 2022 .

[32]  Pat Hanrahan,et al.  Reflection from layered surfaces due to subsurface scattering , 1993, SIGGRAPH.

[33]  Paul A. Beardsley,et al.  Image-based 3D photography using opacity hulls , 2002, ACM Trans. Graph..

[34]  Stephen Lin,et al.  Modeling and rendering of quasi-homogeneous materials , 2005, SIGGRAPH 2005.

[35]  Jon Louis Bentley,et al.  An Algorithm for Finding Best Matches in Logarithmic Expected Time , 1977, TOMS.

[36]  Kristin J. Dana,et al.  Image-based Skin Analysis , 2002 .

[37]  Paul R. Cohen,et al.  Camera Calibration with Distortion Models and Accuracy Evaluation , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[38]  Tim Weyrich,et al.  Robust watermarking of point-sampled geometry , 2004, Proceedings Shape Modeling Applications, 2004..

[39]  Jitendra Malik,et al.  Recovering high dynamic range radiance maps from photographs , 1997, SIGGRAPH '08.

[40]  Thomas Malzbender,et al.  Polynomial texture maps , 2001, SIGGRAPH.

[41]  Hendrik P. A. Lensch,et al.  Efficient, image-based appearance acquisition of real-world objects , 2003 .

[42]  M. Patterson,et al.  Anisotropy of light propagation in human skin , 2000, Physics in medicine and biology.

[43]  Markus H. Gross,et al.  Shape modeling with point-sampled geometry , 2003, ACM Trans. Graph..

[44]  Ling Li,et al.  A multi-layered reflection model of natural human skin , 2001, Proceedings. Computer Graphics International 2001.

[45]  Sunil Arya,et al.  Algorithms for fast vector quantization , 1993, [Proceedings] DCC `93: Data Compression Conference.

[46]  Craig Donner,et al.  Light diffusion in multi-layered translucent materials , 2005, SIGGRAPH 2005.

[47]  Norimichi Tsumura,et al.  Image-based skin color and texture analysis/synthesis by extracting hemoglobin and melanin information in the skin , 2003, ACM Trans. Graph..

[48]  Wei-Chao Chen,et al.  Light field mapping: efficient representation and hardware rendering of surface light fields , 2002, SIGGRAPH.

[49]  Tim Weyrich,et al.  Haptic mediation of risk potentials over a soft tissue data set , 1999 .

[50]  Jos Stam,et al.  Multiple Scattering as a Diffusion Process , 1995, Rendering Techniques.

[51]  Linda G. Shapiro,et al.  View-base Rendering: Visualizing Real Objects from Scanned Range and Color Data , 1997, Rendering Techniques.

[52]  Gladimir V. G. Baranoski,et al.  A Biophysically‐Based Spectral Model of Light Interaction with Human Skin , 2004, Comput. Graph. Forum.

[53]  Paul S. Heckbert,et al.  Fundamentals of Texture Mapping and Image Warping , 1989 .

[54]  Tim Weyrich,et al.  Rendering deformable surface reflectance fields , 2005, IEEE Transactions on Visualization and Computer Graphics.

[55]  Bruno Lévy,et al.  ABF++: fast and robust angle based flattening , 2005, TOGS.

[56]  Kenneth E. Torrance,et al.  A hybrid monte carlo method for accurate and efficient subsurface scattering , 2005, EGSR '05.

[57]  Ligang Liu,et al.  A global Laplacian smoothing approach with feature preservation , 2005, Ninth International Conference on Computer Aided Design and Computer Graphics (CAD-CG'05).

[58]  David Salesin,et al.  A Bayesian approach to digital matting , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[59]  J M Schmitt,et al.  Multilayer model of photon diffusion in skin. , 1990, Journal of the Optical Society of America. A, Optics and image science.

[60]  Markus H. Gross,et al.  Optimized Spatial Hashing for Collision Detection of Deformable Objects , 2003, VMV.

[61]  Marc Alexa,et al.  Point set surfaces , 2001, Proceedings Visualization, 2001. VIS '01..

[62]  Jos Stam,et al.  An Illumination Model for a Skin Layer Bounded by Rough Surfaces , 2001, Rendering Techniques.

[63]  Gavin S. P. Miller,et al.  Lazy Decompression of Surface Light Fields for Precomputed Global Illumination , 1998, Rendering Techniques.

[64]  Athinodoros S. Georghiades,et al.  Recovering 3-D Shape and Reflectance From a Small Number of Photographs , 2003, Rendering Techniques.

[65]  Tim Weyrich,et al.  Post-processing of Scanned 3D Surface Data , 2004, PBG.

[66]  Michael Bosse,et al.  Unstructured lumigraph rendering , 2001, SIGGRAPH.

[67]  Jan J. Koenderink,et al.  The secret of velvety skin , 2003, Machine Vision and Applications.

[68]  Szymon Rusinkiewicz,et al.  A New Change of Variables for Efficient BRDF Representation , 1998, Rendering Techniques.

[69]  D. Levin,et al.  Mesh-Independent Surface Interpolation , 2004 .

[70]  William E. Lorensen,et al.  Marching cubes: A high resolution 3D surface construction algorithm , 1987, SIGGRAPH.

[71]  Hans-Peter Seidel,et al.  Image-Based Reconstruction of Spatially Varying Materials , 2001 .

[72]  Steve Marschner,et al.  Modeling and Rendering for Realistic Facial Animation , 2000, Rendering Techniques.

[73]  Greg Turk,et al.  Re-tiling polygonal surfaces , 1992, SIGGRAPH.

[74]  Michael D. McCool,et al.  Homomorphic factorization of BRDFs for high-performance rendering , 2001, SIGGRAPH.

[75]  Paul Debevec,et al.  Inverse global illumination: Recovering re?ectance models of real scenes from photographs , 1998 .

[76]  Leif Kobbelt,et al.  A survey of point-based techniques in computer graphics , 2004, Comput. Graph..

[77]  J. Dannemiller Spectral reflectance of natural objects: how many basis functions are necessary? , 1992 .

[78]  Katsushi Ikeuchi,et al.  Eigen-texture method: Appearance compression based on 3D model , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[79]  K. Torrance,et al.  Theory for off-specular reflection from roughened surfaces , 1967 .

[80]  Isaac Amidror,et al.  Scattered data interpolation methods for electronic imaging systems: a survey , 2002, J. Electronic Imaging.

[81]  Paul E. Debevec,et al.  Acquiring the reflectance field of a human face , 2000, SIGGRAPH.

[82]  Mark Meyer,et al.  Implicit fairing of irregular meshes using diffusion and curvature flow , 1999, SIGGRAPH.

[83]  Markus Gross,et al.  Point-Based Graphics , 2007 .

[84]  V. Tuchin Tissue Optics: Light Scattering Methods and Instruments for Medical Diagnosis , 2000 .

[85]  Tim Weyrich,et al.  Preoperative Planning and Training Simulation for Risk Reducing Surgery , 1999 .

[86]  Wojciech Matusik,et al.  Acquisition and Rendering of Transparent and Refractive Objects , 2002, Rendering Techniques.

[87]  Wojciech Matusik,et al.  A data-driven reflectance model , 2003, ACM Trans. Graph..

[88]  Katsushi Ikeuchi,et al.  Object shape and reflectance modeling from observation , 1997, SIGGRAPH.

[89]  Matthias Zwicker,et al.  Surface splatting , 2001, SIGGRAPH.

[90]  F. Durand,et al.  Texture design using a simplicial complex of morphable textures , 2005, SIGGRAPH 2005.

[91]  Diego F. Nehab,et al.  Efficiently combining positions and normals for precise 3D geometry , 2005, SIGGRAPH 2005.

[92]  B. Wilson,et al.  A diffusion theory model of spatially resolved, steady-state diffuse reflectance for the noninvasive determination of tissue optical properties in vivo. , 1992, Medical physics.

[93]  Marcus Larsson,et al.  In vivo determination of local skin optical properties and photon path length by use of spatially resolved diffuse reflectance with applications in laser Doppler flowmetry. , 2003, Applied optics.

[94]  Paul E. Green,et al.  Canonical Analysis: An Exposition and Illustrative Application , 1966 .

[95]  Matthias Zwicker,et al.  Surfels: surface elements as rendering primitives , 2000, SIGGRAPH.

[96]  David J. Kriegman,et al.  Helmholtz Stereopsis: Exploiting Reciprocity for Surface Reconstruction , 2002, International Journal of Computer Vision.

[97]  Donald P. Greenberg,et al.  Non-linear approximation of reflectance functions , 1997, SIGGRAPH.

[98]  Gabriel Taubin,et al.  A signal processing approach to fair surface design , 1995, SIGGRAPH.

[99]  Andrew Gardner,et al.  A lighting reproduction approach to live-action compositing , 2002, SIGGRAPH.

[100]  Gregory J. Ward,et al.  Measuring and modeling anisotropic reflection , 1992, SIGGRAPH.

[101]  Sunghee Choi,et al.  A simple algorithm for homeomorphic surface reconstruction , 2000, SCG '00.

[102]  Tim Weyrich,et al.  Mitsubishi Electric Research Laboratories , 2005 .

[103]  Shree K. Nayar,et al.  Reflectance and texture of real-world surfaces , 1999, TOGS.

[104]  Szymon Rusinkiewicz,et al.  Efficient BRDF importance sampling using a factored representation , 2004, SIGGRAPH 2004.

[105]  Leif Kobbelt,et al.  OpenMesh: A Generic and Efficient Polygon Mesh Data Structure , 2002 .

[106]  Andrew Gardner,et al.  Performance geometry capture for spatially varying relighting , 2005, SIGGRAPH '05.

[107]  Jan Kautz,et al.  Interactive rendering with arbitrary BRDFs using separable approximations , 1999, SIGGRAPH '99.

[108]  Tim Weyrich,et al.  A practical structured light acquisition system for point-based geometry and texture , 2005, Proceedings Eurographics/IEEE VGTC Symposium Point-Based Graphics, 2005..

[109]  S. Nayar,et al.  The Appearance of Human Skin , 2005 .

[110]  Tim Weyrich,et al.  Eurographics Symposium on Point-based Graphics (2006) Gpu-based Ray-casting of Quadratic Surfaces , 2022 .

[111]  Hans-Peter Seidel,et al.  An empirical model for heterogeneous translucent objects , 2005, SIGGRAPH '05.

[112]  Baining Guo,et al.  Feature-based light field morphing , 2002, ACM Trans. Graph..

[113]  Kristin J. Dana,et al.  Bidirectional imaging and modeling of skin texture , 2004, IEEE Transactions on Biomedical Engineering.

[114]  Jon Louis Bentley,et al.  Multidimensional binary search trees used for associative searching , 1975, CACM.

[115]  Lance Williams,et al.  Casting curved shadows on curved surfaces , 1978, SIGGRAPH.

[116]  Maria Petrou,et al.  Colour photometric stereo: simultaneous reconstruction of local gradient and colour of rough textured surfaces , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[117]  Neel Joshi,et al.  Noninvasive measurement of scattering anisotropy in turbid materials by nonnormal incident illumination. , 2006, Optics letters.

[118]  Raghu Machiraju,et al.  Finding optimal views for 3D face shape modeling , 2004, Sixth IEEE International Conference on Automatic Face and Gesture Recognition, 2004. Proceedings..

[119]  Jitendra Malik,et al.  Modeling and Rendering Architecture from Photographs: A hybrid geometry- and image-based approach , 1996, SIGGRAPH.

[120]  Richard K. Beatson,et al.  Reconstruction and representation of 3D objects with radial basis functions , 2001, SIGGRAPH.

[121]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[122]  Paul E. Debevec,et al.  A photometric approach to digitizing cultural artifacts , 2001, VAST '01.

[123]  Zhengyou Zhang,et al.  A Flexible New Technique for Camera Calibration , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[124]  Donald P. Greenberg,et al.  A comprehensive physical model for light reflection , 1991, SIGGRAPH.

[125]  Thomas Vetter,et al.  A morphable model for the synthesis of 3D faces , 1999, SIGGRAPH.

[126]  Steve Marschner,et al.  Filling holes in complex surfaces using volumetric diffusion , 2002, Proceedings. First International Symposium on 3D Data Processing Visualization and Transmission.

[127]  Paul E. Debevec,et al.  Optimizing Color Matching in a Lighting Reproduction System for Complex Subject and Illuminant Spectra , 2003, Rendering Techniques.

[128]  Matthias Zwicker,et al.  Pointshop 3D: an interactive system for point-based surface editing , 2002, SIGGRAPH.

[129]  Steve Marschner,et al.  Image-Based BRDF Measurement Including Human Skin , 1999, Rendering Techniques.

[130]  Tamal K. Dey,et al.  Detecting undersampling in surface reconstruction , 2001, SCG '01.

[131]  Peter Schröder,et al.  Interpolating Subdivision for meshes with arbitrary topology , 1996, SIGGRAPH.

[132]  Markus H. Gross,et al.  Efficient simplification of point-sampled surfaces , 2002, IEEE Visualization, 2002. VIS 2002..

[133]  Henrik Wann Jensen,et al.  A rapid hierarchical rendering technique for translucent materials , 2005, SIGGRAPH Courses.

[134]  Hans-Peter Seidel,et al.  Multi-level partition of unity implicits , 2005, SIGGRAPH Courses.

[135]  Andrew Gardner,et al.  Animatable Facial Reflectance Fields , 2004 .

[136]  Raghu Machiraju,et al.  Silhouette-Based 3D Face Shape Recovery , 2003, Graphics Interface.