Mapping and Manipulating Facial Expression

Nonverbal visual cues accompany speech to supplement the meaning of spoken words, signify emotional state, indicate position in discourse, and provide back-channel feedback. This visual information includes head movements, facial expressions and body gestures. In this article we describe techniques for manipulating both verbal and nonverbal facial gestures in video sequences of people engaged in conversation. We are developing a system for use in psychological experiments, where the effects of manipulating individual components of nonverbal visual behavior during live face-to-face conversation can be studied. In particular, the techniques we describe operate in real-time at video frame-rate and the manipulation can be applied so both participants in a conversation are kept blind to the experimental conditions.

[1]  G. Klerman,et al.  Facial Expression and Imagery in Depression: An Electromyographic Study , 1976, Psychosomatic medicine.

[2]  L Sirovich,et al.  Low-dimensional procedure for the characterization of human faces. , 1987, Journal of the Optical Society of America. A, Optics and image science.

[3]  Lawrence Sirovich,et al.  Application of the Karhunen-Loeve Procedure for the Characterization of Human Faces , 1990, IEEE Trans. Pattern Anal. Mach. Intell..

[4]  Lance Williams,et al.  Performance-driven facial animation , 1990, SIGGRAPH.

[5]  M. Turk,et al.  Eigenfaces for Recognition , 1991, Journal of Cognitive Neuroscience.

[6]  W. J. Welsh,et al.  Classification of facial features for recognition , 1991, Proceedings. 1991 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[7]  Demetri Terzopoulos,et al.  Analysis and Synthesis of Facial Image Sequences Using Physical and Anatomical Models , 1993, IEEE Trans. Pattern Anal. Mach. Intell..

[8]  Alex Pentland,et al.  An automatic system for model-based coding of faces , 1995, Proceedings DCC '95 Data Compression Conference.

[9]  Stephen M. Omohundro,et al.  Nonlinear manifold learning for visual speech recognition , 1995, Proceedings of IEEE International Conference on Computer Vision.

[10]  D. Massaro,et al.  Perceiving Talking Faces , 1995 .

[11]  Juergen Luettin,et al.  Speechreading using shape and intensity information , 1996, Proceeding of Fourth International Conference on Spoken Language Processing. ICSLP '96.

[12]  Timothy F. Cootes,et al.  Face Recognition Using Active Appearance Models , 1998, ECCV.

[13]  Timothy F. Cootes,et al.  Active Appearance Models , 1998, ECCV.

[14]  Henrique S. Malvar,et al.  Making Faces , 2019, Topoi.

[15]  Bernard F. Buxton,et al.  Very low bit rate face video compression using linear combination of 2D face views and principal components analysis , 1999, Image Vis. Comput..

[16]  Matthew Turk,et al.  A Morphable Model For The Synthesis Of 3D Faces , 1999, SIGGRAPH.

[17]  Frank J. Bernieri,et al.  The Importance of Nonverbal Cues in Judging Rapport , 1999 .

[18]  D. Shapiro,et al.  Reduced facial expression and social context in major depression: discrepancies between facial muscle activity and self-reported emotion , 2000, Psychiatry Research.

[19]  Jun-yong Noh,et al.  Expression cloning , 2001, SIGGRAPH.

[20]  Hyeong-Seok Ko,et al.  Performance-driven muscle-based facial animation , 2001, Comput. Animat. Virtual Worlds.

[21]  Zicheng Liu,et al.  Expressive expression mapping with ratio images , 2001, SIGGRAPH.

[22]  Harry Shum,et al.  Real-time speech-driven 3D face animation , 2002, Proceedings. First International Symposium on 3D Data Processing Visualization and Transmission.

[23]  Erika Chuang,et al.  Performance Driven Facial Animation using Blendshape Interpolation , 2002 .

[24]  Timothy F. Cootes,et al.  Extraction of Visual Features for Lipreading , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[25]  Thomas Vetter,et al.  Face Recognition Based on Fitting a 3D Morphable Model , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[26]  Gavin C. Cawley,et al.  Towards a low bandwidth talking face using appearance models , 2003, Image Vis. Comput..

[27]  Tomaso A. Poggio,et al.  Reanimating Faces in Images and Video , 2003, Comput. Graph. Forum.

[28]  Jing Xiao,et al.  Real-time combined 2D+3D active appearance models , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[29]  Towards perceptually realistic talking heads: models, methods and McGurk , 2004, APGV '04.

[30]  Jian Yang,et al.  Two-dimensional PCA: a new approach to appearance-based face representation and recognition , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[31]  Simon Baker,et al.  Active Appearance Models Revisited , 2004, International Journal of Computer Vision.

[32]  Gavin C. Cawley,et al.  Near-videorealistic synthetic talking faces: implementation and evaluation , 2004, Speech Commun..

[33]  Toward Perceptually Realistic Talking Heads: Models, Methods, and McGurk , 2005, TAP.

[34]  Tony Ezzat,et al.  Transferable videorealistic speech animation , 2005, SCA '05.

[35]  Hanspeter Pfister,et al.  Face transfer with multilinear models , 2005, ACM Trans. Graph..

[36]  Luiz Velho,et al.  Expression Transfer between Photographs through Multilinear AAM's , 2006, 2006 19th Brazilian Symposium on Computer Graphics and Image Processing.

[37]  Baining Guo,et al.  Geometry-driven photorealistic facial expression synthesis , 2003, IEEE Transactions on Visualization and Computer Graphics.

[38]  Ralph Gross,et al.  Active appearance models with occlusion , 2006, Image Vis. Comput..

[39]  Gérard Bailly,et al.  A new trainable trajectory formation system for facial animation , 2006, ExLing.

[40]  Gérard Bailly,et al.  Intelligibility of natural and 3d-cloned German speech , 2007, AVSP.

[41]  Heloir,et al.  The Uncanny Valley , 2019, The Animation Studies Reader.

[42]  Barry-John Theobald,et al.  A real-time speech-driven talking head using active appearance models , 2007, AVSP.

[43]  Yang Wang,et al.  Enforcing convexity for improved alignment with constrained local models , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.