Face/Off: live facial puppetry

We present a complete integrated system for live facial puppetry that enables high-resolution real-time facial expression tracking with transfer to another person's face. The system utilizes a real-time structured light scanner that provides dense 3D data and texture. A generic template mesh, fitted to a rigid reconstruction of the actor's face, is tracked offline in a training stage through a set of expression sequences. These sequences are used to build a person-specific linear face model that is subsequently used for online face tracking and expression transfer. Even with just a single rigid pose of the target face, convincing real-time facial animations are achievable. The actor becomes a puppeteer with complete and accurate control over a digital face.

[1]  Zhigang Deng,et al.  Computer Facial Animation: A Survey , 2008 .

[2]  Demetri Terzopoulos,et al.  Physically-based facial modelling, analysis, and animation , 1990, Comput. Animat. Virtual Worlds.

[3]  Olaf Schenk,et al.  Solving unsymmetric sparse systems of linear equations with PARDISO , 2002, Future Gener. Comput. Syst..

[4]  Harry Shum,et al.  Face poser: interactive modeling of 3D facial expressions using model priors , 2007, SCA '07.

[5]  Jun-yong Noh,et al.  Expression cloning , 2001, SIGGRAPH.

[6]  Hao Li,et al.  Global Correspondence Optimization for Non‐Rigid Registration of Depth Scans , 2008, Comput. Graph. Forum.

[7]  Michael M. Cohen,et al.  Modeling Coarticulation in Synthetic Visual Speech , 1993 .

[8]  Henning Biermann,et al.  Recovering non-rigid 3D shape from image streams , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[9]  Thomas Vetter,et al.  A morphable model for the synthesis of 3D faces , 1999, SIGGRAPH.

[10]  Byung-Uck Kim,et al.  Real-time data driven deformation using kernel canonical correlation analysis , 2008, SIGGRAPH 2008.

[11]  Pieter Peers,et al.  Facial performance synthesis using deformation-driven polynomial displacement maps , 2008, SIGGRAPH 2008.

[12]  Zhigang Deng,et al.  Animating blendshape faces by cross-mapping motion capture data , 2006, I3D '06.

[13]  Marc Levoy,et al.  Real-time 3D model acquisition , 2002, ACM Trans. Graph..

[14]  Lance Williams,et al.  Performance-driven facial animation , 1990, SIGGRAPH Courses.

[15]  Baining Guo,et al.  Geometry-driven photorealistic facial expression synthesis , 2003, IEEE Transactions on Visualization and Computer Graphics.

[16]  Christoph Bregler,et al.  Analysis, Synthesis, and Retargeting of Facial Expressions , 2004 .

[17]  Pieter Peers,et al.  Facial performance synthesis using deformation-driven polynomial displacement maps , 2008, SIGGRAPH Asia '08.

[18]  Berthold K. P. Horn,et al.  Determining Optical Flow , 1981, Other Conferences.

[19]  Keith Waters,et al.  Computer Facial Animation, Second Edition , 1996 .

[20]  Marc Levoy,et al.  Efficient variants of the ICP algorithm , 2001, Proceedings Third International Conference on 3-D Digital Imaging and Modeling.

[21]  Li Zhang,et al.  Spacetime faces: high resolution capture for modeling and animation , 2004, SIGGRAPH 2004.

[22]  Christoph Bregler,et al.  Mood swings: expressive speech animation , 2005, TOGS.

[23]  Luc Van Gool,et al.  Fast 3D Scanning with Automatic Motion Compensation , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[24]  Parke,et al.  Parameterized Models for Facial Animation , 1982, IEEE Computer Graphics and Applications.

[25]  Jovan Popovic,et al.  Deformation transfer for triangle meshes , 2004, ACM Trans. Graph..

[26]  Tomaso A. Poggio,et al.  Reanimating Faces in Images and Video , 2003, Comput. Graph. Forum.

[27]  Byung-Uck Kim,et al.  Real-time data driven deformation using kernel canonical correlation analysis , 2008, ACM Trans. Graph..

[28]  Jing Xiao,et al.  Vision-based control of 3D facial animation , 2003, SCA '03.

[29]  Dimitris N. Metaxas,et al.  Optical Flow Constraints on Deformable Models with Applications to Face Tracking , 2000, International Journal of Computer Vision.

[30]  L Sirovich,et al.  Low-dimensional procedure for the characterization of human faces. , 1987, Journal of the Optical Society of America. A, Optics and image science.

[31]  Pascal Müller,et al.  Animation pipeline: realistic speech based on observed 3D face dynamics , 2004 .

[32]  Hanspeter Pfister,et al.  Face transfer with multilinear models , 2005, SIGGRAPH 2005.

[33]  Frederick I. Parke,et al.  Computer generated animation of faces , 1972, ACM Annual Conference.

[34]  Berthold K. P. Horn,et al.  Determining Optical Flow , 1981, Other Conferences.

[35]  David Salesin,et al.  Resynthesizing facial animation through 3D model-based tracking , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[36]  Norman I. Badler,et al.  Animating facial expressions , 1981, SIGGRAPH '81.

[37]  Jovan Popović,et al.  Deformation transfer for triangle meshes , 2004, SIGGRAPH 2004.

[38]  Lance Williams,et al.  Human face project , 2005, SIGGRAPH Courses.

[39]  Olaf Schenk,et al.  Solving unsymmetric sparse systems of linear equations with PARDISO , 2004, Future Gener. Comput. Syst..

[40]  R. Fisher.  THE USE OF MULTIPLE MEASUREMENTS IN TAXONOMIC PROBLEMS , 1936 .

[41]  Olga Sorkine-Hornung,et al.  On Linear Variational Surface Deformation Methods , 2008, IEEE Transactions on Visualization and Computer Graphics.

[42]  Sami Romdhani,et al.  Optimal Step Nonrigid ICP Algorithms for Surface Registration , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[43]  Mathieu Desbrun,et al.  Learning controls for blend shape based realistic facial animation , 2003, SIGGRAPH '03.

[44]  Zoran Popovic,et al.  The space of human body shapes: reconstruction and parameterization from range scans , 2003, ACM Trans. Graph..

[45]  Keith Waters,et al.  Computer facial animation , 1996 .

[46]  Parag Havaldar.  Sony Pictures Imageworks , 2006, SIGGRAPH Courses.

[47]  Markus H. Gross,et al.  Pose-space animation and transfer of facial details , 2008, SCA '08.

[48]  Atsuto Maki,et al.  Mimicking video: real-time morphable 3D model fitting , 2003, VRST '03.

[49]  Ronald Fedkiw,et al.  Eurographics/ Acm Siggraph Symposium on Computer Animation (2006) Simulating Speech with a Physics-based Facial Muscle Model , 2022 .

[50]  Helmut Pottmann,et al.  Registration of point cloud data from a geometric optimization perspective , 2004, SGP '04.

[51]  Lance Williams,et al.  Performance-driven facial animation , 1990, SIGGRAPH.

[52]  Hanspeter Pfister,et al.  Face transfer with multilinear models , 2005, ACM Trans. Graph..

[53]  Matthew Turk,et al.  A Morphable Model For The Synthesis Of 3D Faces , 1999, SIGGRAPH.