Dynamic, expressive speech animation from a single mesh

In this work we present a method for human face animation which allows us to generate animations for a novel person given just a single mesh of their face. These animations can be of arbitrary text and may include emotional expressions. We build a multilinear model from data which encapsulates the variation in dynamic face motions over changes in identity, expression, and over different texts. We then describe a synthesis method consisting of a phoneme planning and a blending stage which uses this model as a base and attempts to preserve both face shape and dynamics given a novel text and an emotion at each point in time.

[1]  P. Ekman,et al.  Facial action coding system: a technique for the measurement of facial movement , 1978 .

[2]  Parke,et al.  Parameterized Models for Facial Animation , 1982, IEEE Computer Graphics and Applications.

[3]  Keith Waters,et al.  A muscle model for animation three-dimensional facial expression , 1987, SIGGRAPH.

[4]  Lance Williams,et al.  Performance-driven facial animation , 1990, SIGGRAPH.

[5]  Demetri Terzopoulos,et al.  Realistic modeling for facial animation , 1995, SIGGRAPH.

[6]  Christoph Bregler,et al.  Video Rewrite: Driving Visual Speech with Audio , 1997, SIGGRAPH.

[7]  L. Lathauwer,et al.  Signal Processing based on Multilinear Algebra , 1997 .

[8]  Matthew Brand,et al.  Voice puppetry , 1999, SIGGRAPH.

[9]  Matthew Turk,et al.  A Morphable Model For The Synthesis Of 3D Faces , 1999, SIGGRAPH.

[10]  Demetri Terzopoulos,et al.  Multilinear Analysis of Image Ensembles: TensorFaces , 2002, ECCV.

[11]  Jing Xiao,et al.  Vision-based control of 3D facial animation , 2003, SCA '03.

[12]  Tomaso A. Poggio,et al.  Reanimating Faces in Images and Video , 2003, Comput. Graph. Forum.

[13]  Demetri Terzopoulos,et al.  TensorTextures: multilinear image-based rendering , 2004, ACM Trans. Graph..

[14]  Michael Jones,et al.  Multidimensional Morphable Models: A Framework for Representing and Matching Object Classes , 2004, International Journal of Computer Vision.

[15]  Li Zhang,et al.  Spacetime faces: high resolution capture for modeling and animation , 2004, SIGGRAPH 2004.

[16]  Tony Ezzat,et al.  Trainable videorealistic speech animation , 2002, Sixth IEEE International Conference on Automatic Face and Gesture Recognition, 2004. Proceedings..

[17]  M. Alex O. Vasilescu,et al.  TensorTextures: multilinear image-based rendering , 2004, SIGGRAPH 2004.

[18]  Shmuel Peleg,et al.  Seamless Image Stitching in the Gradient Domain , 2004, ECCV.

[19]  Frédéric H. Pighin,et al.  Expressive speech-driven facial animation , 2005, TOGS.

[20]  F. Durand,et al.  Texture design using a simplicial complex of morphable textures , 2005, ACM Trans. Graph..

[21]  Narendra Ahuja,et al.  Out-of-core tensor approximation of multi-dimensional matrices of visual data , 2005, ACM Trans. Graph..

[22]  Pascal Müller,et al.  Realistic speech animation based on observed 3-D face dynamics , 2005 .

[23]  Scott A. King,et al.  Creating speech-synchronized animation , 2005, IEEE Transactions on Visualization and Computer Graphics.

[24]  Tony Ezzat,et al.  Transferable videorealistic speech animation , 2005, SCA '05.

[25]  Hanspeter Pfister,et al.  Face transfer with multilinear models , 2005, ACM Trans. Graph..

[26]  Christoph Bregler,et al.  Mood swings: expressive speech animation , 2005, TOGS.

[27]  John P. Lewis,et al.  Synthesizing speech animation by learning compact speech co-articulation models , 2005, International 2005 Computer Graphics.

[28]  N. Ahuja,et al.  Out-of-core tensor approximation of multi-dimensional matrices of visual data , 2005, SIGGRAPH 2005.

[29]  John P. Lewis,et al.  Expressive Facial Animation Synthesis by Learning Speech Coarticulation and Expression Spaces , 2006, IEEE Transactions on Visualization and Computer Graphics.

[30]  Ronald Fedkiw,et al.  Eurographics/ Acm Siggraph Symposium on Computer Animation (2006) Simulating Speech with a Physics-based Facial Muscle Model , 2022 .

[31]  John P. Lewis,et al.  Facial motion retargeting , 2006, SIGGRAPH Courses.