Animating blendshape faces by cross-mapping motion capture data

Animating 3D faces to achieve compelling realism is a challenging task in the entertainment industry. Previously proposed face transfer approaches generally require a high-quality animated source face in order to transfer its motion to new 3D faces. In this work, we present a semi-automatic technique to directly animate popularized 3D blendshape face models by mapping facial motion capture data spaces to 3D blendshape face spaces. After sparse markers on the face of a human subject are captured by motion capture systems while a video camera is simultaneously used to record his/her front face, then we carefully select a few motion capture frames and accompanying video frames as reference mocap-video pairs. Users manually tune blendshape weights to perceptually match the animated blendshape face models with reference facial images (the reference mocap-video pairs) in order to create reference mocap-weight pairs. Finally, the Radial Basis Function (RBF) regression technique is used to map any new facial motion capture frame to blendshape weights based on the reference mocap-weight pairs. Our results demonstrate that this technique is efficient to animate blendshape face models, while offering its generality and flexiblity.

[1]  Thomas Vetter,et al.  A morphable model for the synthesis of 3D faces , 1999, SIGGRAPH.

[2]  Dimitris N. Metaxas,et al.  The integration of optical flow and deformable models with applications to human face shape and motion estimation , 1996, Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[3]  David Salesin,et al.  Resynthesizing facial animation through 3D model-based tracking , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[4]  John P. Lewis,et al.  Reducing blendshape interference by selected motion attenuation , 2005, I3D '05.

[5]  Jovan Popović,et al.  Deformation transfer for triangle meshes , 2004, SIGGRAPH 2004.

[6]  Tomaso A. Poggio,et al.  Reanimating Faces in Images and Video , 2003, Comput. Graph. Forum.

[7]  John P. Lewis,et al.  Synthesizing speech animation by learning compact speech co-articulation models , 2005, International 2005 Computer Graphics.

[8]  Lance Williams,et al.  Performance-driven facial animation , 1990, SIGGRAPH Courses.

[9]  Martin D. Buhmann,et al.  Radial Basis Functions , 2021, Encyclopedia of Mathematical Geosciences.

[10]  Lance Williams,et al.  Performance-driven facial animation , 1990, SIGGRAPH.

[11]  Keith Waters,et al.  Computer Facial Animation, Second Edition , 1996 .

[12]  Hanspeter Pfister,et al.  Face transfer with multilinear models , 2005, ACM Trans. Graph..

[13]  Matthew Turk,et al.  A Morphable Model For The Synthesis Of 3D Faces , 1999, SIGGRAPH.

[14]  Tony Ezzat,et al.  Trainable videorealistic speech animation , 2002, Sixth IEEE International Conference on Automatic Face and Gesture Recognition, 2004. Proceedings..

[15]  Sung Yong Shin,et al.  An example-based approach for facial expression cloning , 2003, SCA '03.

[16]  Jun-yong Noh,et al.  Expression cloning , 2001, SIGGRAPH.

[17]  Nadia Magnenat-Thalmann,et al.  Visyllable Based Speech Animation , 2003, Comput. Graph. Forum.

[18]  M. B. Stegmann,et al.  A Brief Introduction to Statistical Shape Analysis , 2002 .

[19]  Sung Yong Shin,et al.  Scattered Data Interpolation with Multilevel B-Splines , 1997, IEEE Trans. Vis. Comput. Graph..

[20]  Thaddeus Beier,et al.  Feature-based image metamorphosis , 1992, SIGGRAPH.

[21]  Jing Xiao,et al.  Vision-based control of 3D facial animation , 2003, SCA '03.

[22]  Christoph Bregler,et al.  Video Rewrite: Driving Visual Speech with Audio , 1997, SIGGRAPH.

[23]  Ronald A. Cole,et al.  Accurate automatic visible speech synthesis of arbitrary 3D models based on concatenation of diviseme motion capture data: Research Articles , 2004 .

[24]  Alex Pentland,et al.  Modeling, tracking and interactive animation of faces and heads//using input from video , 1996, Proceedings Computer Animation '96.

[25]  Ronald Fedkiw,et al.  Automatic determination of facial muscle activations from sparse motion capture marker data , 2005, SIGGRAPH '05.

[26]  Eddie Kohler,et al.  Real-time speech motion synthesis from recorded motions , 2004, SCA '04.

[27]  Hyeong-Seok Ko,et al.  Analysis and synthesis of facial expressions with hand-generated muscle actuation basis , 2001, Proceedings Computer Animation 2001. Fourteenth Conference on Computer Animation (Cat. No.01TH8596).

[28]  David Salesin,et al.  Synthesizing realistic facial expressions from photographs , 1998, SIGGRAPH.

[29]  Matthew Brand,et al.  Voice puppetry , 1999, SIGGRAPH.

[30]  Gregory M. Nielson,et al.  Scattered data modeling , 1993, IEEE Computer Graphics and Applications.

[31]  Ronald Fedkiw,et al.  Automatic determination of facial muscle activations from sparse motion capture marker data , 2005, ACM Trans. Graph..

[32]  Lance Williams,et al.  Motion signal processing , 1995, SIGGRAPH.

[33]  Jovan Popovic,et al.  Deformation transfer for triangle meshes , 2004, ACM Trans. Graph..

[34]  John P. Lewis,et al.  Pose Space Deformation: A Unified Approach to Shape Interpolation and Skeleton-Driven Deformation , 2000, SIGGRAPH.

[35]  Pertti Roivainen,et al.  3-D Motion Estimation in Model-Based Facial Image Coding , 1993, IEEE Trans. Pattern Anal. Mach. Intell..

[36]  Ulrich Neumann,et al.  Gesture Driven Facial Animation , 2002 .

[37]  Ronald A. Cole,et al.  Accurate automatic visible speech synthesis of arbitrary 3D models based on concatenation of diviseme motion capture data , 2004, Comput. Animat. Virtual Worlds.

[38]  Zhigang Deng,et al.  Natural head motion synthesis driven by acoustic prosodic features , 2005, Comput. Animat. Virtual Worlds.

[39]  Hyeong-Seok Ko,et al.  Analysis and synthesis of facial expressions with hand-generated muscle actuation basis , 2005, SIGGRAPH Courses.

[40]  Remington Scott.  Sparking life: notes on the performance capture sessions for the Lord of the Rings: the Two Towers , 2003, COMG.