Head-Mounted Photometric Stereo for Performance Capture

Head-mounted cameras are an increasingly important tool for capturing facial performances to drive virtual characters. They provide a fixed, unoccluded view of the face, useful for observing motion capture dots or as input to video analysis. However, the 2D imagery captured with these systems is typically affected by ambient light and generally fails to record subtle 3D shape changes as the face performs. We have developed a system that augments a head-mounted camera with LED-based photometric stereo. The system allows observation of the face independent of the ambient light and generates per-pixel surface normals so that the performance is recorded dynamically in 3D. The resulting data can be used for facial relighting or as better input to machine learning algorithms for driving an animated face.

[1]  Timothy F. Cootes,et al.  Active Appearance Models , 1998, ECCV.

[2]  Luc Van Gool,et al.  Face/Off: live facial puppetry , 2009, SCA '09.

[3]  Daniel Cremers,et al.  Anisotropic Huber-L1 Optical Flow , 2009, BMVC.

[4]  Thomas Malzbender,et al.  Surface enhancement using real-time photometric stereo and reflectance transformation , 2006, EGSR '06.

[5]  Moshe Ben-Ezra,et al.  Photometric Stereo for Dynamic Surface Orientations , 2010, ECCV.

[6]  Marc Levoy,et al.  Real-time 3D model acquisition , 2002, ACM Trans. Graph..

[7]  Yasuyuki Matsushita,et al.  High-quality shape from multi-view stereo and shading under general illumination , 2011, CVPR 2011.

[8]  Ronen Basri,et al.  Photometric stereo with general, unknown lighting , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[9]  Andrew Gardner,et al.  Performance relighting and reflectance transformation with time-multiplexed illumination , 2005, ACM Trans. Graph..

[10]  Pieter Peers,et al.  Rapid Acquisition of Specular and Diffuse Normal Maps from Polarized Spherical Gradient Illumination , 2007 .

[11]  Song Zhang,et al.  High-resolution, real-time three-dimensional shape measurement , 2006 .

[12]  Martin Klaudiny,et al.  High-Detail 3D Capture and Non-sequential Alignment of Facial Performance , 2012, 2012 Second International Conference on 3D Imaging, Modeling, Processing, Visualization & Transmission.

[13]  Wan-Chun Ma,et al.  Comprehensive Facial Performance Capture , 2011, Comput. Graph. Forum.

[14]  Wojciech Matusik,et al.  Multi-scale capture of facial geometry and motion , 2007, ACM Trans. Graph..

[15]  Richard Szeliski,et al.  A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[16]  John P. Lewis,et al.  Universal capture: image-based facial animation for "The Matrix Reloaded" , 2003, SIGGRAPH '03.

[17]  Nancy S. Pollard,et al.  Real-time gradient-domain painting , 2008, ACM Trans. Graph..

[18]  Pieter Peers,et al.  Facial performance synthesis using deformation-driven polynomial displacement maps , 2008, SIGGRAPH Asia '08.

[19]  Szymon Rusinkiewicz,et al.  Spacetime Stereo: A Unifying Framework for Depth from Triangulation , 2005, IEEE Trans. Pattern Anal. Mach. Intell..

[20]  George Vogiatzis,et al.  Self-calibrating a real-time monocular 3 d facial capture system , 2010 .

[21]  Robert J. Woodham,et al.  Photometric method for determining surface orientation from multiple images , 1980 .

[22]  Graham Fyffe,et al.  Single-shot photometric stereo by spectral multiplexing , 2010, 2011 IEEE International Conference on Computational Photography (ICCP).

[23]  Mubarak Shah,et al.  Integration of shape from shading and stereo , 1995, Pattern Recognit..

[24]  Horst Bischof,et al.  Motion estimation with non-local total variation regularization , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[25]  Diego F. Nehab,et al.  Efficiently combining positions and normals for precise 3D geometry , 2005, SIGGRAPH 2005.

[26]  Lance Williams,et al.  Performance-driven facial animation , 1990, SIGGRAPH Courses.

[27]  Steven M. Seitz,et al.  Spacetime faces , 2004, ACM Trans. Graph..

[28]  Thomas Vetter,et al.  A morphable model for the synthesis of 3D faces , 1999, SIGGRAPH.

[29]  Hao Li,et al.  Realtime performance-based facial animation , 2011, ACM Trans. Graph..

[30]  Jan Kautz,et al.  Capturing multiple illumination conditions using time and color multiplexing , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[31]  Richard Szeliski,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, International Journal of Computer Vision.

[32]  Roberto Cipolla,et al.  Overcoming Shadows in 3-Source Photometric Stereo , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[33]  Thabo Beeler,et al.  High-quality single-shot capture of facial geometry , 2010, ACM Trans. Graph..