Performance-driven hand-drawn animation

We present a novel method for generating performance-driven, "hand-drawn" animation in real-time. Given an annotated set of hand-drawn faces for various expressions, our algorithm performs multi-way morphs to generate real-time animation that mimics the expressions of a user. Our system consists of a vision-based tracking component and a rendering component. Together, they form an animation system that can be used in a variety of applications, including teleconferencing, multi-user virtual worlds, compressed instructional videos, and consumer-oriented animation kits.This paper describes our algorithms in detail and illustrates the potential for this work in a teleconferencing application. Experience with our implementation suggests that there are several advantages to our hand-drawn characters over other alternatives: (1) flexibility of animation style; (2) increased compression of expression information; and (3) masking of errors made by the face tracking system that are distracting in photorealistic animations.

[1]  David Salesin,et al.  Synthesizing realistic facial expressions from photographs , 1998, SIGGRAPH.

[2]  Fabio Lavagetto,et al.  Synthetic and hybrid imaging in the HUMANOID and VIDAS projects , 1996, Proceedings of 3rd IEEE International Conference on Image Processing.

[3]  Peter C. Litwinowicz,et al.  Facial Animation by Spatial Mapping , 1991 .

[4]  Alex Pentland,et al.  An automatic system for model-based coding of faces , 1995, Proceedings DCC '95 Data Compression Conference.

[5]  Stephen E. Librande.  Example-based character drawing , 1992 .

[6]  Keith Waters,et al.  Computer facial animation , 1996 .

[7]  Matthew Brand,et al.  Voice puppetry , 1999, SIGGRAPH.

[8]  David Salesin,et al.  Multiperspective panoramas for cel animation , 1997, SIGGRAPH.

[9]  Paul Haeberli,et al.  Paint by numbers: abstract image representations , 1990, SIGGRAPH.

[10]  Tony Ezzat,et al.  Facial analysis and synthesis using image-based models , 1996, Proceedings of the Second International Conference on Automatic Face and Gesture Recognition.

[11]  F. Thomas,et al.  The illusion of life : Disney animation , 1981 .

[12]  R. Koenen.  Coding of Moving Pictures and Audio , 1996 .

[13]  Kiyoharu Aizawa,et al.  Analysis and synthesis of facial image sequences in model-based image coding , 1994, IEEE Trans. Circuits Syst. Video Technol..

[14]  Michele Covell,et al.  Eigen-points: control-point location using principal component analyses , 1996, Proceedings of the Second International Conference on Automatic Face and Gesture Recognition.

[15]  David Salesin,et al.  Computer-generated watercolor , 1997, SIGGRAPH.

[16]  Gert Vegter,et al.  In handbook of discrete and computational geometry , 1997 .

[17]  Kentaro Toyama,et al.  Prolegomena for Robust Face Tracking , 1998 .

[18]  Lance Williams,et al.  Animating images with drawings , 1994, SIGGRAPH.

[19]  Michael J. Black,et al.  Tracking and recognizing rigid and non-rigid facial motions using local parametric models of image motion , 1995, Proceedings of IEEE International Conference on Computer Vision.

[20]  Peter M. Grant,et al.  Automatic frame fitting for semantic-based moving image coding using a facial code-book , 1996, 1996 8th European Signal Processing Conference (EUSIPCO 1996).

[21]  Keith Waters,et al.  Computer Facial Animation, Second Edition , 1996 .

[22]  Krzysztof,et al.  INTERNATIONAL ORGANISATION FOR STANDARDISATION , 2006 .

[23]  David Salesin,et al.  Orientable textures for image-based pen-and-ink illustration , 1997, SIGGRAPH.

[24]  Hans Peter Graf,et al.  Robust face feature analysis for automatic speechreading and character animation , 1996, Proceedings of the Second International Conference on Automatic Face and Gesture Recognition.

[25]  Sung Yong Shin,et al.  Polymorph: Morphing Among Multiple Images , 1998, IEEE Computer Graphics and Applications.

[26]  David Salesin,et al.  Resynthesizing facial animation through 3D model-based tracking , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[27]  Matthew Turk,et al.  A Morphable Model For The Synthesis Of 3D Faces , 1999, SIGGRAPH.

[28]  Alex Pentland,et al.  3D modeling and tracking of human lip motions , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[29]  Paul A. Beardsley,et al.  Computer Vision for Interactive Computer Graphics , 1998, IEEE Computer Graphics and Applications.

[30]  Alex Pentland,et al.  Modeling, tracking and interactive animation of faces and heads//using input from video , 1996, Proceedings Computer Animation '96.

[31]  Tomaso Poggio,et al.  Example Based Image Analysis and Synthesis , 1993 .

[32]  Michael Isard,et al.  Active Contours: The Application of Techniques from Graphics, Vision, Control Theory and Statistics to Visual Tracking of Shapes in Motion , 2000 .

[33]  F. A. Seiler,et al.  Numerical Recipes in C: The Art of Scientific Computing , 1989 .

[34]  Demetri Terzopoulos,et al.  Snakes: Active contour models , 2004, International Journal of Computer Vision.

[35]  James O. Normile,et al.  Tracking subspace representations of face images , 1994, Proceedings of ICASSP '94. IEEE International Conference on Acoustics, Speech and Signal Processing.

[36]  Lance Williams,et al.  Performance-driven facial animation , 1990, SIGGRAPH.

[37]  Michael M. Cohen,et al.  Real-time analysis-synthesis and intelligibility of talking faces , 1994, SSW.

[38]  P. Grant,et al.  Accurate automatic frame fitting for semantic-based moving image coding using a facial code-book , 1996, Proceedings of 3rd IEEE International Conference on Image Processing.

[39]  Pertti Roivainen,et al.  3-D Motion Estimation in Model-Based Facial Image Coding , 1993, IEEE Trans. Pattern Anal. Mach. Intell..

[40]  Duncan Rowland,et al.  Manipulating facial appearance through shape and color , 1995, IEEE Computer Graphics and Applications.

[41]  Thaddeus Beier,et al.  Feature-based image metamorphosis , 1992, SIGGRAPH.

[42]  Timothy F. Cootes,et al.  A unified approach to coding and interpreting face images , 1995, Proceedings of IEEE International Conference on Computer Vision.

[43]  Fernando Pereira,et al.  MPEG-4 facial animation technology: survey, implementation, and results , 1999, IEEE Trans. Circuits Syst. Video Technol..

[44]  Alex Pentland,et al.  Parametrized structure from motion for 3D adaptive feedback tracking of faces , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[45]  Sebastian Toelg,et al.  Towards an Example-Based Image Compression Architecture for Video-Conferencing , 1994 .

[46]  Heekuck Oh,et al.  Neural Networks for Pattern Recognition , 1993, Adv. Comput..

[47]  Demetri Terzopoulos,et al.  Realistic modeling for facial animation , 1995, SIGGRAPH.

[48]  Henrique S. Malvar,et al.  Making Faces , 2019, Topoi.

[49]  Adam Finkelstein,et al.  Texture mapping for cel animation , 1998, SIGGRAPH.

[50]  Chil-Woo Lee,et al.  Automatic recognition of human facial expressions , 1995, Proceedings of IEEE International Conference on Computer Vision.

[51]  Gregory M. Nielson,et al.  Scattered data modeling , 1993, IEEE Computer Graphics and Applications.

[52]  Demetri Terzopoulos,et al.  Analysis and Synthesis of Facial Image Sequences Using Physical and Anatomical Models , 1993, IEEE Trans. Pattern Anal. Mach. Intell..

[53]  W. Press,et al.  Numerical Recipes in C++: The Art of Scientific Computing (2nd edn)1 Numerical Recipes Example Book (C++) (2nd edn)2 Numerical Recipes Multi-Language Code CD ROM with LINUX or UNIX Single-Screen License Revised Version3 , 2003 .

[54]  Norman I. Badler,et al.  Animating facial expressions , 1981, SIGGRAPH '81.

[55]  David Salesin,et al.  Comic Chat , 1996, SIGGRAPH.

[56]  Keith Waters,et al.  A muscle model for animation three-dimensional facial expression , 1987, SIGGRAPH.