Animating Speech in Games

Realism in games is constantly improving with increased computing power available to games and as game players demand more visual realism. Therefore, facial animation and particularly animated speech, is becoming more prevalent in games. We present a survey of facial animation techniques suitable for 3D computer games. We break our discussion into two areas: modeling and animation. To model a face, a method of representing the geometry is combined with a parameterization that is used to specify a new shape for that geometry. Changing the shape over time will create animation, and we discuss methods for animating expressions as well as achieving lip-synchronized speech.

[1]  Gary Faigin,et al.  The Artistʼs Complete Guide to Facial Expression , 1990 .

[2]  Kiyoharu Aizawa,et al.  Real-time facial action image synthesis system driven by speech and text , 1990, Other Conferences.

[3]  Thoms M. Levergood,et al.  DEC face: an automatic lip-synchronization algorithm for synthetic faces , 1993 .

[4]  W. Larrabee,et al.  A finite element model of skin deformation. III. The finite element model , 1986, The Laryngoscope.

[5]  Norman I. Badler,et al.  Animating facial expressions , 1981, SIGGRAPH '81.

[6]  Irfan Essa,et al.  Analysis, interpretation and synthesis of facial expressions , 1995 .

[7]  David J. Sturman,et al.  Computer Puppetry , 1998, IEEE Computer Graphics and Applications.

[8]  Keith Waters,et al.  A muscle model for animation three-dimensional facial expression , 1987, SIGGRAPH.

[9]  Zhigang Deng,et al.  Eurographics/ Acm Siggraph Symposium on Computer Animation (2006) Efase: Expressive Facial Animation Synthesis and Editing with Phoneme-isomap Controls , 2022 .

[10]  Brendan McCane,et al.  Language-driven nonverbal communication in a bilingual conversational agent , 2003, Proceedings 11th IEEE International Workshop on Program Comprehension.

[11]  Norman I. Badler,et al.  Eyes alive , 2002, ACM Trans. Graph..

[12]  Demetri Terzopoulos,et al.  Physically-based facial modelling, analysis, and animation , 1990, Comput. Animat. Virtual Worlds.

[13]  Wayne E. Carlson,et al.  Simulating the structure and dynamics of human hair: Modelling, rendering and animation , 1991, Comput. Animat. Virtual Worlds.

[14]  Hans-Peter Seidel,et al.  Speech Synchronization for Physics-Based Facial Animation , 2002, WSCG.

[15]  Scott A. King,et al.  Use and Re-use of Facial Motion Capture Data , 2003, VVG.

[16]  N. Magnenat-Thalmann,et al.  Synthetic actors in computer-generated 3D films , 1990 .

[17]  Irene Albrecht,et al.  Automatic Generation of Non-Verbal Facial Expressions from Speech , 2002 .

[18]  Masahiro Tomono,et al.  Story driven animation , 1987, CHI 1987.

[19]  Alice Wang,et al.  Assembling an expressive facial animation system , 2007, Sandbox '07.

[20]  Leonard T. Bruton,et al.  Lip synchronization in 3-D model based coding for video-conferencing , 1995, Proceedings of ISCAS'95 - International Symposium on Circuits and Systems.

[21]  Nadia Magnenat-Thalmann,et al.  Visyllable Based Speech Animation , 2003, Comput. Graph. Forum.

[22]  Scott A. King,et al.  A 3D parametric tongue model for animated speech , 2001, Comput. Animat. Virtual Worlds.

[23]  Scott A. King,et al.  A facial model and animation techniques for animated speech , 2001 .

[24]  Steve Marschner,et al.  Light scattering from human hair fibers , 2003, ACM Trans. Graph..

[25]  Daniel Thalmann,et al.  Models and Techniques in Computer Animation , 2014, Computer Animation Series.

[26]  Justine Cassell,et al.  BEAT: the Behavior Expression Animation Toolkit , 2001, Life-like characters.

[27]  Marie-Luce Viaud,et al.  Facial animation with wrinkles , 1992 .

[28]  Hans-Peter Seidel,et al.  Head shop: generating animated head models with anatomical structure , 2002, SCA '02.

[29]  Lance Williams,et al.  Performance-driven facial animation , 1990, SIGGRAPH Courses.

[30]  P. Ekman.  Telling lies: clues to deceit in the marketplace , 1985 .

[31]  Frederic I. Parke,et al.  A parametric model for human faces. , 1974 .

[32]  P. Ekman.  Unmasking The Face , 1975 .

[33]  Steve Marschner,et al.  Modeling and Rendering for Realistic Facial Animation , 2000, Rendering Techniques.

[34]  Zhigang Deng,et al.  Animating blendshape faces by cross-mapping motion capture data , 2006, I3D '06.

[35]  Hans B. Sieburg,et al.  Medicine Meets Virtual Reality: Health Care in the Information Age , 1996 .

[36]  Catherine Pelachaud,et al.  Rule-Structured Facial Animation System , 1993, IJCAI.

[37]  Bruce W. Weide,et al.  A computer system for simulating human facial expression , 1989 .

[38]  Tomoyuki Nishita,et al.  Animating Hair with Loosely Connected Particles , 2003, Comput. Graph. Forum.

[39]  Tony Ezzat,et al.  Videorealistic talking faces: a morphing approach , 1997, AVSP.

[40]  Scott A. King,et al.  A Muscle-based 3D Parametric Lip Model for Speech-Synchronized Facial Animation , 2000, DEFORM/AVATARS.

[41]  Frédéric H. Pighin,et al.  Expressive speech-driven facial animation , 2005, TOGS.

[42]  Raymond D. Kent,et al.  Coarticulation in recent speech production models , 1977 .

[43]  Brian Wyvill,et al.  Speech and expression: a computer solution to face animation , 1986 .

[44]  Wojciech Matusik,et al.  Multi-scale capture of facial geometry and motion , 2007, ACM Trans. Graph..

[45]  Tim Weyrich,et al.  Analysis of human faces using a measurement-based skin reflectance model , 2006, ACM Trans. Graph..

[46]  Stefan Van Niekerk,et al.  Playable universal capture , 2006, SIGGRAPH Sketches.

[47]  Markus H. Gross,et al.  Emotion Editing using Finite Elements , 1998, Comput. Graph. Forum.

[48]  Keith Waters,et al.  Computer facial animation , 1996 .

[49]  C. Pelachaud.  Communication and coarticulation in facial animation , 1992 .

[50]  Hans-Peter Seidel,et al.  MIMIC - A Language for Specifying Facial Animations , 2004, WSCG.

[51]  Peter C. Litwinowicz,et al.  Facial Animation by Spatial Mapping , 1991 .

[52]  Stephen Michael Platt,et al.  A structural model of the human face (graphics, animation, object representation) , 1985 .

[53]  Jason Osipa.  Stop Staring: Facial Modeling and Animation Done Right , 2003 .

[54]  H Chiyokura,et al.  Surgical simulation of facial paralysis. , 1996, Studies in health technology and informatics.

[55]  Hans-Peter Seidel,et al.  Geometry-based Muscle Modeling for Facial Animation , 2001, Graphics Interface.

[56]  Tony Ezzat,et al.  MikeTalk: a talking facial display based on morphing visemes , 1998, Proceedings Computer Animation '98 (Cat. No.98EX169).

[57]  Steve Marschner,et al.  A practical model for subsurface light transport , 2001, SIGGRAPH.

[58]  Michael M. Cohen,et al.  Modeling Coarticulation in Synthetic Visual Speech , 1993 .

[59]  Anders Löfqvist,et al.  Speech as Audible Gestures , 1990 .

[60]  Demetri Terzopoulos,et al.  Modelling and animating faces using scanned data , 1991, Comput. Animat. Virtual Worlds.

[61]  Monique Nahas,et al.  Animation of a B-Spline figure , 1988, The Visual Computer.

[62]  Manjula Patel,et al.  FACES: Facial Animation, Construction and Editing System , 1991, Eurographics.

[63]  Nadia Magnenat-Thalmann,et al.  Simulation of static and dynamic wrinkles of skin , 1996, Proceedings Computer Animation '96.

[64]  N. Badler,et al.  Linguistic Issues in Facial Animation , 1991 .

[65]  Bill Fleming,et al.  Animating Facial Features & Expressions , 1998 .

[66]  Markus H. Gross,et al.  Simulating facial surgery using finite element models , 1996, SIGGRAPH.

[67]  Mark Steedman,et al.  Animated conversation: rule-based generation of facial expression, gesture & spoken intonation for multiple conversational agents , 1994, SIGGRAPH.

[68]  Ulrich Neumann,et al.  Interactive multiresolution hair modeling and editing , 2002, SIGGRAPH.

[69]  Ronald Fedkiw,et al.  Automatic determination of facial muscle activations from sparse motion capture marker data , 2005, ACM Trans. Graph..

[70]  Daniel Thalmann,et al.  Abstract muscle action procedures for human face animation , 1988, The Visual Computer.

[71]  Bertrand Le Goff,et al.  Automatic modeling of coarticulation in text-to-visual speech synthesis , 1997, EUROSPEECH.

[72]  Michael T. Ghiselin,et al.  Darwin and Facial Expression: A Century of Research in Review.@@@Darwin on Man: A Psychological Study of Scientific Creativity. , 1974 .

[73]  Brian Guenter,et al.  A System for Simulating Human Facial Expression , 1989 .

[74]  Irfan A. Essa,et al.  Real-time Photo-Realistic Physically Based Rendering of Fine Scale Human Skin Structure , 2001, Rendering Techniques.

[75]  Hans-Peter Seidel,et al.  Real-time rendering of human hair using programmable graphics hardware , 2004 .

[76]  Clea T. Waite,et al.  The facial action control editor, face : a parametric facial expression editor for computer generated animation , 1989 .