Example-Based facial animation for blend shape interpolation

This paper presents a system for example-based character animation using shape interpolation (blend shapes) generated from tracked offline facial feature. First, the subject was recorded expressing various facial expressions (happy, sad, fear, anger, surprise). The recordings were then processed and marked on specific places (feature points) on the face where certain expressions would have the most significant change (the eyes and mouth area). The facial movements, or control parameters, of the human subject are used as movement parameter for the freeform deformation blend shape interpolation.

[1]  Matthew Brand,et al.  Voice puppetry , 1999, SIGGRAPH.

[2]  Markus H. Gross,et al.  Pose-space animation and transfer of facial details , 2008, SCA '08.

[3]  Demetri Terzopoulos,et al.  Physically-based facial modelling, analysis, and animation , 1990, Comput. Animat. Virtual Worlds.

[4]  Gwen Littlewort,et al.  Dynamics of Facial Expression Extracted Automatically from Video , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[5]  Garrison W. Cottrell,et al.  Representing Face Images for Emotion Classification , 1996, NIPS.

[6]  Pieter Peers,et al.  Facial performance synthesis using deformation-driven polynomial displacement maps , 2008, SIGGRAPH 2008.

[7]  John P. Lewis,et al.  Reducing blendshape interference by selected motion attenuation , 2005, I3D '05.

[8]  Narendra M. Patel,et al.  Parametric Facial Expression Synthesis and Animation , 2010 .

[9]  Zhen Liu,et al.  A Robust Blob Recognition and Tracking Method in Vision-Based Multi-touch Technique , 2008, 2008 IEEE International Symposium on Parallel and Distributed Processing with Applications.

[10]  M. Otaduy,et al.  Multi-scale capture of facial geometry and motion , 2007, ACM Trans. Graph..

[11]  Matti Pietikäinen,et al.  Boosted multi-resolution spatiotemporal descriptors for facial expression recognition , 2009, Pattern Recognit. Lett..

[12]  Ana Belén Moreno,et al.  Differential optical flow applied to automatic facial expression recognition , 2011, Neurocomputing.

[13]  Richard E. Parent,et al.  Layered construction for deformable animated characters , 1989, SIGGRAPH.

[14]  Hazim Kemal Ekenel,et al.  A common framework for real-time emotion recognition and facial action unit detection , 2011, CVPR 2011 WORKSHOPS.

[15]  Jun-yong Noh,et al.  Animated deformations with radial basis functions , 2000, VRST '00.

[16]  Zhiwei Zhu,et al.  Dynamic Facial Expression Analysis and Synthesis With MPEG-4 Facial Animation Parameters , 2008, IEEE Transactions on Circuits and Systems for Video Technology.

[17]  Frederic I. Parke,et al.  A parametric model for human faces. , 1974 .

[18]  Alex Pentland,et al.  Modeling, tracking and interactive animation of faces and heads//using input from video , 1996, Proceedings Computer Animation '96.

[19]  F. Hara,et al.  Facial interaction between animated 3D face robot and human beings , 1997, 1997 IEEE International Conference on Systems, Man, and Cybernetics. Computational Cybernetics and Simulation.

[20]  Wojciech Matusik,et al.  Multi-scale capture of facial geometry and motion , 2007, ACM Trans. Graph..

[21]  Gwen Littlewort,et al.  Recognizing facial expression: machine learning and application to spontaneous behavior , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[22]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[23]  Daniel Thalmann,et al.  Joint-dependent local deformations for hand animation and object grasping , 1989 .

[24]  Leif Kobbelt,et al.  Character animation from 2D pictures and 3D motion data , 2007, TOGS.

[25]  Tsuneya Kurihara,et al.  Modeling deformable human hands from medical images , 2004, SCA '04.

[26]  Tony Ezzat,et al.  Trainable videorealistic speech animation , 2002, Sixth IEEE International Conference on Automatic Face and Gesture Recognition, 2004. Proceedings..

[27]  Joachim M. Buhmann,et al.  Pairwise Data Clustering by Deterministic Annealing , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[28]  Myung Jin Chung,et al.  Development of a Facial Expression Imitation System , 2006, 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[29]  Henrique S. Malvar,et al.  Making Faces , 2019, Topoi.

[30]  Hiromitsu Hama.  An Accurate Method for Finding the Control Points of Bezier Curves , 1997 .

[31]  Frederick I. Parke,et al.  Computer gernerated animation of faces , 1998 .

[32]  Jun-yong Noh,et al.  Expression cloning , 2001, SIGGRAPH.

[33]  Lifeng Shang,et al.  Nonparametric discriminant HMM and application to facial expression recognition , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[34]  Michael J. Lyons,et al.  Automatic Classification of Single Facial Images , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[35]  Zhigang Deng,et al.  Animating blendshape faces by cross-mapping motion capture data , 2006, I3D '06.

[36]  Shyan-Ming Yuan,et al.  Vision-Based Finger Detection, Tracking, and Event Identification Techniques for Multi-Touch Sensing and Display Systems , 2011, Sensors.

[37]  Roberto Brunelli,et al.  Face Recognition: Features Versus Templates , 1993, IEEE Trans. Pattern Anal. Mach. Intell..

[38]  Gwen Littlewort,et al.  Automatic Recognition of Facial Actions in Spontaneous Expressions , 2006, J. Multim..

[39]  Michael M. Cohen,et al.  Modeling Coarticulation in Synthetic Visual Speech , 1993 .

[40]  Mohammed Yeasin,et al.  Recognition of facial expressions and measurement of levels of interest from video , 2006, IEEE Transactions on Multimedia.

[41]  Enrique Muñoz,et al.  Recognising facial expressions in video sequences , 2007, Pattern Analysis and Applications.

[42]  Marian Stewart Bartlett,et al.  Facial expression recognition using Gabor motion energy filters , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops.

[43]  John P. Lewis,et al.  Real‐Time Weighted Pose‐Space Deformation on the GPU , 2006, Comput. Graph. Forum.

[44]  Gwen Littlewort,et al.  Dynamics of Facial Expression Extracted Automatically from Video , 2004, CVPR Workshops.

[45]  Bobby Bodenheimer,et al.  Synthesis and evaluation of linear motion transitions , 2008, TOGS.

[46]  Aggelos K. Katsaggelos,et al.  Automatic facial expression recognition using facial animation parameters and multistream HMMs , 2006, IEEE Transactions on Information Forensics and Security.

[47]  Takeo Kanade,et al.  The Extended Cohn-Kanade Dataset (CK+): A complete dataset for action unit and emotion-specified expression , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops.

[48]  M. Pauly,et al.  Example-based facial rigging , 2010, ACM Trans. Graph..

[49]  P. Ekman,et al.  Facial action coding system: a technique for the measurement of facial movement , 1978 .

[50]  Daniel Thalmann,et al.  Abstract muscle action procedures for human face animation , 1988, The Visual Computer.

[51]  Taehyun Rhee,et al.  Real-time facial animation from live video tracking , 2011, SCA '11.

[52]  Maja Pantic,et al.  Expert system for automatic analysis of facial expressions , 2000, Image Vis. Comput..

[53]  Christoph Bregler,et al.  Video Rewrite: Driving Visual Speech with Audio , 1997, SIGGRAPH.

[54]  Larry S. Davis,et al.  Recognizing Human Facial Expressions From Long Image Sequences Using Optical Flow , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[55]  Byung-Uck Kim,et al.  Real-time data driven deformation using kernel canonical correlation analysis , 2008, ACM Trans. Graph..

[56]  Jing Xiao,et al.  Vision-based control of 3D facial animation , 2003, SCA '03.

[57]  Shaogang Gong,et al.  Facial expression recognition based on Local Binary Patterns: A comprehensive study , 2009, Image Vis. Comput..

[58]  John P. Lewis,et al.  Pose Space Deformation: A Unified Approach to Shape Interpolation and Skeleton-Driven Deformation , 2000, SIGGRAPH.

[59]  Keith Waters,et al.  A muscle model for animation three-dimensional facial expression , 1987, SIGGRAPH.

[60]  Yunus Saatci,et al.  Cascaded classification of gender and facial expression using active appearance models , 2006, 7th International Conference on Automatic Face and Gesture Recognition (FGR06).

[61]  Weimin Huang,et al.  Attentive Behavior Detection by Non-Linear Head Pose Embedding and Mapping , 2005, 2005 IEEE 7th Workshop on Multimedia Signal Processing.

[62]  Pieter Peers,et al.  Facial performance synthesis using deformation-driven polynomial displacement maps , 2008, SIGGRAPH Asia '08.

[63]  Mathieu Desbrun,et al.  Learning controls for blend shape based realistic facial animation , 2003, SIGGRAPH '03.

[64]  Alex Pentland,et al.  LAFTER: lips and face real time tracker , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[65]  Michael J. Black,et al.  Recognizing Facial Expressions in Image Sequences Using Local Parameterized Models of Image Motion , 1997, International Journal of Computer Vision.

[66]  Demetri Terzopoulos,et al.  Analysis and Synthesis of Facial Image Sequences Using Physical and Anatomical Models , 1993, IEEE Trans. Pattern Anal. Mach. Intell..

[67]  Lance Williams,et al.  Performance-driven facial animation , 1990, SIGGRAPH.

[68]  Daniel Thalmann,et al.  Simulation of Facial Muscle Actions Based on Rational Free Form Deformations , 1992, Comput. Graph. Forum.

[69]  Alex Pentland,et al.  Coding, Analysis, Interpretation, and Recognition of Facial Expressions , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[70]  Frederick I. Parke,et al.  Computer generated animation of faces , 1972, ACM Annual Conference.

[71]  Demetri Terzopoulos,et al.  Realistic modeling for facial animation , 1995, SIGGRAPH.

[72]  Zoran Popovic,et al.  Articulated body deformation from range scan data , 2002, SIGGRAPH.