A Framework for Capture and Synthesis of High Resolution Facial Geometry and Performance

We present a framework that captures and synthesizes high resolution facial geometry and performance. In order to capture highly detailed surface structures, a theory of fast normal recovery using spherical gradient illumination patterns is presented to estimate surface normal maps of an object from either its diffuse or specular reflectance, simultaneously from any viewpoints. We show that the normal map from specular reflectance yields the best record of detailed surface shape, which can be used for geometry enhancement. Moreover, the normal map from the diffuse reflectance is able to produce a good approximation of subsurface scattering. Based on the theory, two systems are developed to capture high resolution facial geometry of a static face or dynamic facial performance. The static face scanning system consists of a spherical illumination device, two singlelens reflex (SLR) cameras and a video projector. The spherical illumination device is used to cast spherical gradient patterns onto the subject. The captured spherical gradient images are then turned into surface normals of the subject. The two cameras and one projector are used to build a structured-light-assisted two-view stereo system, which acquires a moderate resolution geometry of the subject. We then use the acquired specular normal map to enhance the initial geometry based on an optimization process. To further analyze how facial geometry deforms during performance, we build another facial performance capture system, which is analogous to the previous face scanning system, but employs two high-speed video cameras and a high-speed projector. The system is able to capture 30 facial geometry measurements per second. A novel method named polynomial displacement maps is presented to cooperate motion capture with real-time face scans, so that realistic facial deformation can be modeled and synthesized. Finally, we present a real-time relighting algorithm based on spherical wavelets for rendering realistic faces under modern GPU architecture.

[1]  Demetri Terzopoulos,et al.  Physically-based facial modelling, analysis, and animation , 1990, Comput. Animat. Virtual Worlds.

[2]  Harry Shum,et al.  Eurographics Symposium on Rendering (2004) All-frequency Precomputed Radiance Transfer for Glossy Objects , 2022 .

[3]  Lawrence B. Wolff,et al.  Using polarization to separate reflection components , 1989, Proceedings CVPR '89: IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[4]  Jaakko Lehtinen,et al.  Matrix radiance transfer , 2003, I3D '03.

[5]  Hayden Landis,et al.  Production-Ready Global Illumination , 2004 .

[6]  L. Mahadevan,et al.  Geometry and physics of wrinkling. , 2003, Physical review letters.

[7]  Demetri Terzopoulos,et al.  Realistic modeling for facial animation , 1995, SIGGRAPH.

[8]  Mathieu Desbrun,et al.  Learning controls for blend shape based realistic facial animation , 2003, SIGGRAPH '03.

[9]  Peter Schröder,et al.  Spherical wavelets: efficiently representing functions on the sphere , 1995, SIGGRAPH.

[10]  Katsushi Ikeuchi,et al.  Object shape and reflectance modeling from observation , 1997, SIGGRAPH.

[11]  Masashi Baba,et al.  Estimating roughness parameters of an object's surface from real images , 2004, SIGGRAPH '04.

[12]  Christopher Oat.  Animated wrinkle maps , 2007, SIGGRAPH '07.

[13]  Peter Schröder,et al.  Spherical Wavelets: Texture Processing , 1995, Rendering Techniques.

[14]  Steve Marschner,et al.  Image-Based BRDF Measurement Including Human Skin , 1999, Rendering Techniques.

[15]  Georges-Pierre Bonneau,et al.  Optimal triangular Haar bases for spherical data , 1999, Proceedings Visualization '99 (Cat. No.99CB37067).

[16]  David Salesin,et al.  Animating pictures with stochastic motion textures , 2005, ACM Trans. Graph..

[17]  Hans-Peter Seidel,et al.  3D acquisition of mirroring objects using striped patterns , 2005, Graph. Model..

[18]  Zhengyou Zhang,et al.  A Flexible New Technique for Camera Calibration , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[19]  Robert J. Woodham,et al.  Photometric method for determining surface orientation from multiple images , 1980 .

[20]  Rui Wang,et al.  All-frequency relighting of glossy objects , 2006, TOGS.

[21]  Wojciech Matusik,et al.  A data-driven reflectance model , 2003, ACM Trans. Graph..

[22]  John P. Lewis,et al.  Pose Space Deformation: A Unified Approach to Shape Interpolation and Skeleton-Driven Deformation , 2000, SIGGRAPH.

[23]  Peter-Pike J. Sloan,et al.  Clustered principal components for precomputed radiance transfer , 2003, ACM Trans. Graph..

[24]  Andrew Gardner,et al.  Animatable Facial Reflectance Fields , 2004 .

[25]  David Salesin,et al.  Environment matting and compositing , 1999, SIGGRAPH.

[26]  Zen-Chung Shih,et al.  All-frequency precomputed radiance transfer using spherical radial basis functions and clustered tensor approximation , 2006, ACM Trans. Graph..

[27]  David Salesin,et al.  Synthesizing realistic facial expressions from photographs , 1998, SIGGRAPH.

[28]  Thomas Brox,et al.  High Accuracy Optical Flow Estimation Based on a Theory for Warping , 2004, ECCV.

[29]  Takeo Kanade,et al.  Determining shape and reflectance of hybrid surfaces by photometric sampling , 1989, IEEE Trans. Robotics Autom..

[30]  Ronald Fedkiw,et al.  Automatic determination of facial muscle activations from sparse motion capture marker data , 2005, ACM Trans. Graph..

[31]  Wojciech Matusik,et al.  Multi-scale capture of facial geometry and motion , 2007, ACM Trans. Graph..

[32]  Pat Hanrahan,et al.  All-frequency shadows using non-linear wavelet lighting approximation , 2003, ACM Trans. Graph..

[33]  Jan Kautz,et al.  Precomputed radiance transfer for real-time rendering in dynamic, low-frequency lighting environments , 2002 .

[34]  Hans-Peter Seidel,et al.  Image-based reconstruction of spatial appearance and geometric detail , 2003, TOGS.

[35]  Shree K. Nayar,et al.  Separation of Reflection Components Using Color and Polarization , 1997, International Journal of Computer Vision.

[36]  Gregory M. Nielson,et al.  Haar wavelets over triangular domains with applications to multiresolution models for flow over a sphere , 1997, Proceedings. Visualization '97 (Cat. No. 97CB36155).

[37]  Andrew Gardner,et al.  Performance geometry capture for spatially varying relighting , 2005, SIGGRAPH '05.

[38]  Thomas Malzbender,et al.  Surface enhancement using real-time photometric stereo and reflectance transformation , 2006, EGSR '06.

[39]  Rui Wang,et al.  All-frequency interactive relighting of translucent objects with single and multiple scattering , 2005, ACM Trans. Graph..

[40]  David Salesin,et al.  Environment matting extensions: towards higher accuracy and real-time capture , 2000, SIGGRAPH.

[41]  Wojciech Matusik,et al.  A statistical model for synthesis of detailed facial geometry , 2006, SIGGRAPH 2006.

[42]  David J. Kriegman,et al.  Beyond Lambert: reconstructing specular surfaces using color , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[43]  L. B. Wolff.  Material classification and separation of reflection components using polarization/radiometric information , 1989 .

[44]  Daniel Cohen-Or,et al.  Bilateral mesh denoising , 2003 .

[45]  Frédo Durand,et al.  View-dependent precomputed light transport using nonlinear Gaussian function approximations , 2006, I3D '06.

[46]  Li Zhang,et al.  Rapid shape acquisition using color structured light and multi-pass dynamic programming , 2002, Proceedings. First International Symposium on 3D Data Processing Visualization and Transmission.

[47]  Peter-Pike J. Sloan,et al.  Local, deformable precomputed radiance transfer , 2005, ACM Trans. Graph..

[48]  Thomas Malzbender,et al.  Polynomial texture maps , 2001, SIGGRAPH.

[49]  Takeo Kanade,et al.  Shape from interreflections , 2004, International Journal of Computer Vision.

[50]  Andrew Gardner,et al.  Performance relighting and reflectance transformation with time-multiplexed illumination , 2005, ACM Trans. Graph..

[51]  Rui Wang,et al.  Eurographics Symposium on Rendering (2004) All-frequency Relighting of Non-diffuse Objects Using Separable Brdf Approximation , 2022 .

[52]  Peter-Pike J. Sloan.  Normal mapping for precomputed radiance transfer , 2006, I3D '06.

[53]  Katsushi Ikeuchi,et al.  Determining Surface Orientations of Specular Surfaces by Using the Photometric Stereo Method , 1981, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[54]  Paul E. Debevec,et al.  Acquiring the reflectance field of a human face , 2000, SIGGRAPH.

[55]  Jovan Popovic,et al.  Inverse kinematics for reduced deformable models , 2006, SIGGRAPH '06.

[56]  P. Debevec,et al.  A Reflective Light Stage , 2006 .

[57]  Henrique S. Malvar,et al.  Making Faces , 2019, Topoi.

[58]  Hans-Peter Seidel,et al.  Mesostructure from Specularity , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[59]  Frederick I. Parke,et al.  Computer gernerated animation of faces , 1998 .

[60]  Marc Levoy,et al.  Real-time 3D model acquisition , 2002, ACM Trans. Graph..

[61]  Ravi Ramamoorthi,et al.  Reflectance sharing: image-based rendering from a sparse set of images , 2005, EGSR '05.

[62]  Hans-Peter Seidel,et al.  Polarization and Phase-Shifting for 3D Scanning of Translucent Objects , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[63]  Maria Petrou,et al.  The 4-Source Photometric Stereo Technique for Three-Dimensional Surfaces in the Presence of Highlights and Shadows , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[64]  Andrew Gardner,et al.  Linear light source reflectometry , 2003, ACM Trans. Graph..

[65]  Jan Kautz,et al.  Fast Arbitrary BRDF Shading for Low-Frequency Lighting Using Spherical Harmonics , 2002, Rendering Techniques.

[66]  Karsten Schlüns,et al.  Photometric Stereo for Non-Lambertian Surfaces Using Color Information , 1993, CAIP.

[67]  Szymon Rusinkiewicz,et al.  Efficiently combining positions and normals for precise 3D geometry , 2005, ACM Trans. Graph..

[68]  Irfan A. Essa,et al.  Real-time Photo-Realistic Physically Based Rendering of Fine Scale Human Skin Structure , 2001, Rendering Techniques.

[69]  Andrew Chi-Sing Leung,et al.  Data compression with spherical wavelets and wavelets for the image-based relighting , 2004, Comput. Vis. Image Underst..

[70]  Pat Hanrahan,et al.  Frequency space environment map rendering , 2002, SIGGRAPH.

[71]  Frederick I. Parke,et al.  Computer generated animation of faces , 1972, ACM Annual Conference.

[72]  M. Gross,et al.  Analysis of human faces using a measurement-based skin reflectance model , 2006, ACM Trans. Graph..

[73]  Brian A. Barsky,et al.  Reconstructing curved surfaces from specular reflection patterns using spline surface fitting of normals , 1996, SIGGRAPH.

[74]  Li Zhang,et al.  Spacetime faces: high resolution capture for modeling and animation , 2004, SIGGRAPH 2004.

[75]  Song Zhang,et al.  High-resolution, real-time three-dimensional shape measurement , 2006 .

[76]  Szymon Rusinkiewicz,et al.  Spacetime stereo: a unifying framework for depth from triangulation , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[77]  Charles T. Loop,et al.  Computing rectifying homographies for stereo vision , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[78]  P. Hanrahan,et al.  Triple product wavelet integrals for all-frequency relighting , 2004, SIGGRAPH 2004.

[79]  Lance Williams,et al.  Performance-driven facial animation , 1990, SIGGRAPH.

[80]  Pat Hanrahan,et al.  An efficient representation for irradiance environment maps , 2001, SIGGRAPH.

[81]  Wojciech Matusik,et al.  A statistical model for synthesis of detailed facial geometry , 2006, ACM Trans. Graph..