Near-instant capture of high-resolution facial geometry and reflectance

Modeling realistic human characters is frequently done using 3D recordings of the shape and appearance of real people, often across a set of different facial expressions to build blendshape facial models. Believable characters that cross the "Uncanny Valley" require high-quality geometry, texture maps, reflectance properties, and surface detail at the level of skin pores and fine wrinkles. Unfortunately, there has not yet been a technique for recording such datasets that is near-instantaneous and low-cost. While some facial capture techniques are instantaneous and inexpensive [Beeler et al. 2010], these do not generally provide lighting-independent texture maps, specular reflectance information, or high-resolution surface normal detail for relighting. In contrast, techniques which use multiple photographs from spherical lighting setups [Ghosh et al. 2011] do capture such reflectance properties, at the expense of longer capture times and complicated custom equipment.

[1]  N. R. Bartlett,et al.  Latency of the blink reflex and stimulus intensity J , 1967 .

[2]  Robert J. Woodham,et al.  Photometric Stereo: A Reflectance Map Technique For Determining Surface Orientation From Image Intensity , 1979, Optics & Photonics.

[3]  S. Zucker,et al.  Shape-from-shading on a cloudy day , 1994 .

[4]  Gabriel Taubin,et al.  Appying Shape from Lighting Variation to Bump Map Capture , 1997, Rendering Techniques.

[5]  Paul E. Debevec,et al.  Acquiring the reflectance field of a human face , 2000, SIGGRAPH.

[6]  Marc Levoy,et al.  Real-time 3D model acquisition , 2002, ACM Trans. Graph..

[7]  Athinodoros S. Georghiades,et al.  Recovering 3-D Shape and Reflectance From a Small Number of Photographs , 2003, Rendering Techniques.

[8]  Todd E. Zickler,et al.  Helmholtz Stereopsis: Exploiting Reciprocity for Surface Reconstruction , 2002, International Journal of Computer Vision.

[9]  Shree K. Nayar,et al.  Separation of Reflection Components Using Color and Polarization , 1997, International Journal of Computer Vision.

[10]  Steven M. Seitz,et al.  Spacetime faces , 2004, ACM Trans. Graph..

[11]  Steven M. Seitz,et al.  Example-based photometric stereo: shape reconstruction with general, varying BRDFs , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  Jessica C. Ramella-Roman,et al.  Out-of-plane polarimetric imaging of skin: surface and subsurface effects , 2005, SPIE BiOS.

[13]  David J. Kriegman,et al.  Beyond Lambert: reconstructing specular surfaces using color , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[14]  Szymon Rusinkiewicz,et al.  Spacetime Stereo: A Unifying Framework for Depth from Triangulation , 2005, IEEE Trans. Pattern Anal. Mach. Intell..

[15]  Frédéric H. Pighin,et al.  Synthesizing realistic facial expressions from photographs , 2005, SIGGRAPH Courses.

[16]  Diego F. Nehab,et al.  Efficiently combining positions and normals for precise 3D geometry , 2005, SIGGRAPH 2005.

[17]  Steven M. Seitz,et al.  Shape and Spatially-Varying BRDFs from Photometric Stereo , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[18]  Ravi Ramamoorthi,et al.  Reflectance sharing: predicting appearance from a sparse set of images of a known shape , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[19]  Hans-Peter Seidel,et al.  Mesostructure from Specularity , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[20]  David J. Kriegman,et al.  Specularity Removal in Images and Videos: A PDE Approach , 2006, ECCV.

[21]  David J. Kriegman,et al.  Color Subspaces as Photometric Invariants , 2006, CVPR.

[22]  Song Zhang,et al.  High-resolution, real-time three-dimensional shape measurement , 2006 .

[23]  M. Gross,et al.  Analysis of human faces using a measurement-based skin reflectance model , 2006, ACM Trans. Graph..

[24]  Pieter Peers,et al.  Rapid Acquisition of Specular and Diffuse Normal Maps from Polarized Spherical Gradient Illumination , 2007 .

[25]  Björn Stenger,et al.  Non-rigid Photometric Stereo with Colored Lights , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[26]  Abhijeet Ghosh,et al.  Practical modeling and acquisition of layered facial reflectance , 2008, SIGGRAPH 2008.

[27]  Caroline Jay,et al.  A perceptually validated model for surface depth hallucination , 2008, ACM Trans. Graph..

[28]  Jean Ponce,et al.  Dense 3D motion capture for human faces , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[29]  Jean Ponce,et al.  Accurate, Dense, and Robust Multiview Stereopsis , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[30]  Derek Bradley,et al.  High resolution passive facial performance capture , 2010, ACM Trans. Graph..

[31]  Thabo Beeler,et al.  High-quality single-shot capture of facial geometry , 2010, ACM Trans. Graph..

[32]  Wan-Chun Ma,et al.  The Digital Emily Project: Achieving a Photorealistic Digital Actor , 2010, IEEE Computer Graphics and Applications.

[33]  Paul E. Debevec,et al.  Multiview face capture using polarized spherical gradient illumination , 2011, ACM Trans. Graph..

[34]  Hans-Peter Seidel,et al.  Shading-based dynamic shape refinement from multi-view video under general illumination , 2011, 2011 International Conference on Computer Vision.

[35]  Martin Klaudiny,et al.  High-Detail 3D Capture and Non-sequential Alignment of Facial Performance , 2012, 2012 Second International Conference on 3D Imaging, Modeling, Processing, Visualization & Transmission.

[36]  Hans-Peter Seidel,et al.  Lightweight binocular facial performance capture under uncontrolled lighting , 2012, ACM Trans. Graph..

[37]  Paul A. Beardsley,et al.  Coupled 3D reconstruction of sparse facial hair and skin , 2012, ACM Trans. Graph..

[38]  Szymon Rusinkiewicz,et al.  Structure-aware hair capture , 2013, ACM Trans. Graph..

[39]  Derek Bradley,et al.  High-quality capture of eyes , 2014, ACM Trans. Graph..