Communication and Automatic Interpretation of Affect from Facial Expressions

The objective of this chapter is to introduce the reader to the recent advances in computer processing of facial expressions and communicated affect. Human facial expressions have evolved in tandem with human face recognition abilities, and show remarkable consistency across cultures. Consequently, it is rewarding to review the main traits of face recognition in humans, as well as consolidated research on the categorization of facial expressions. The bulk of the chapter focuses on the main trends in computer analysis of facial expressions, sketching out the main algorithms and exposing computational considerations for different settings. We then look at some recent applications and promising new projects to give the reader a realistic view of what to expect from this technology now and in near future.

[1]  H. Bülthoff,et al.  The contribution of different facial regions to the recognition of conversational expressions. , 2008, Journal of vision.

[2]  Hasan Demirel,et al.  Facial Expression Recognition Using 3D Facial Feature Distances , 2007, ICIAR.

[3]  Andrei Popescu-Belis,et al.  Multimodal Signal Processing , 2012 .

[4]  Michael G. Strintzis,et al.  Bilinear Models for 3-D Face and Facial Expression Recognition , 2008, IEEE Transactions on Information Forensics and Security.

[5]  L. Obler,et al.  Right hemisphere emotional perception: evidence across multiple channels. , 1998, Neuropsychology.

[6]  M. Tarr,et al.  Activation of the middle fusiform 'face area' increases with expertise in recognizing novel objects , 1999, Nature Neuroscience.

[7]  Cynthia Whissell,et al.  THE DICTIONARY OF AFFECT IN LANGUAGE , 1989 .

[8]  P. Ekman,et al.  Unmasking the Face: A Guide to Recognizing Emotions From Facial Expressions , 1975 .

[9]  Alan Hanjalic,et al.  Affective video content representation and modeling , 2005, IEEE Transactions on Multimedia.

[10]  Kang Ryoung Park,et al.  A comparative study of facial appearance modeling methods for active appearance models , 2009, Pattern Recognit. Lett..

[11]  Maja Pantic,et al.  Non-rigid registration using free-form deformations for recognition of facial actions and their temporal dynamics , 2008, 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition.

[12]  P. Ekman.  Facial expression and emotion. , 1993, The American psychologist.

[13]  Qiang Ji,et al.  Active and dynamic information fusion for facial expression understanding from image sequences , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  Maja Pantic,et al.  Fully Automatic Facial Action Unit Detection and Temporal Analysis , 2006, 2006 Conference on Computer Vision and Pattern Recognition Workshop (CVPRW'06).

[15]  Gwen Littlewort,et al.  Automatic coding of facial expressions displayed during posed and genuine pain , 2009, Image Vis. Comput..

[16]  Qiang Ji,et al.  A probabilistic framework for modeling and real-time monitoring human fatigue , 2006, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[17]  George N. Votsis,et al.  Emotion recognition in human-computer interaction , 2001, IEEE Signal Process. Mag..

[18]  Andrew Ortony,et al.  The Cognitive Structure of Emotions , 1988 .

[19]  Peter Robinson,et al.  Perception of emotional expressions in different representations using facial feature points , 2009, 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops.

[20]  Matthew Turk,et al.  A Morphable Model For The Synthesis Of 3D Faces , 1999, SIGGRAPH.

[21]  Arman Savran,et al.  3D Face Recognition Benchmarks on the Bosphorus Database with Focus on Facial Expressions , 2008, BIOID.

[22]  Maja Pantic,et al.  Social signal processing: Survey of an emerging domain , 2009, Image Vis. Comput..

[23]  Michael J. Lyons,et al.  Automatic Classification of Single Facial Images , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[24]  Hatice Gunes,et al.  How to distinguish posed from spontaneous smiles using geometric features , 2007, ICMI '07.

[25]  S. Baron-Cohen,et al.  Mind Reading: The Interactive Guide to Emotions , 2003 .

[26]  Tamie Salter.  A Need for Flexible Robotic Devices , .

[27]  Andrew J. Calder,et al.  PII: S0042-6989(01)00002-5 , 2001 .

[28]  P. Ekman.  Emotions revealed , 2004, BMJ.

[29]  Gwen Littlewort,et al.  Drowsy Driver Detection Through Facial Movement Analysis , 2007, ICCV-HCI.

[30]  D. Perrett,et al.  Dissociable neural responses to facial expressions of sadness and anger. , 1999, Brain : a journal of neurology.

[31]  Timothy F. Cootes,et al.  Active Appearance Models , 1998, ECCV.

[32]  K. Scherer.  Psychological models of emotion. , 2000 .

[33]  Paul A. Viola,et al.  Robust Real-Time Face Detection , 2001, International Journal of Computer Vision.

[34]  Gwen Littlewort,et al.  Toward Practical Smile Detection , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[35]  A. Young,et al.  Understanding face recognition. , 1986, British journal of psychology.

[36]  Marc Garbey,et al.  Interacting with human physiology , 2007, Comput. Vis. Image Underst..

[37]  Cynthia Breazeal,et al.  Affective Learning — A Manifesto , 2004 .

[38]  Jeffrey F. Cohn,et al.  The Timing of Facial Motion in posed and Spontaneous Smiles , 2003, Int. J. Wavelets Multiresolution Inf. Process..

[39]  M. Hasselmo,et al.  The role of expression and identity in the face-selective responses of neurons in the temporal visual cortex of the monkey , 1989, Behavioural Brain Research.

[40]  Jing Xiao,et al.  Automatic analysis and recognition of brow actions and head motion in spontaneous facial behavior , 2004, 2004 IEEE International Conference on Systems, Man and Cybernetics (IEEE Cat. No.04CH37583).

[41]  Karen L. Schmidt,et al.  Human facial expressions as adaptations: Evolutionary questions in facial expression research. , 2001, American journal of physical anthropology.

[42]  Catherine Pelachaud,et al.  From brows to trust: evaluating embodied conversational agents , 2004 .

[43]  Rosalind W. Picard,et al.  Affective Computing and Autism , 2006, Annals of the New York Academy of Sciences.

[44]  P. Ekman,et al.  Unmasking the face : a guide to recognizing emotions from facial clues , 1975 .

[45]  Terrence J Sejnowski,et al.  Foundations for a New Science of Learning , 2009, Science.

[46]  Takeo Kanade,et al.  Comprehensive database for facial expression analysis , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[47]  Alex Pentland,et al.  Human-Centred Intelligent Human-Computer Interaction (HCI2): how far are we from attaining it? , 2008, Int. J. Auton. Adapt. Commun. Syst..

[48]  Koray Balci.  Xface: MPEG-4 based open source toolkit for 3D Facial Animation , 2004, AVI.

[49]  Cynthia Breazeal,et al.  Affective Interaction between Humans and Robots , 2001, ECAL.

[50]  Fred Nicolls,et al.  Locating Facial Features with an Extended Active Shape Model , 2008, ECCV.

[51]  E. C. Grant,et al.  Human Facial Expression , 1969 .

[52]  L. F. Barrett,et al.  Handbook of Emotions , 1993 .

[53]  Hatice Gunes,et al.  From the Lab to the real world: affect recognition using multiple cues and modalities , 2008 .

[54]  Beat Fasel,et al.  Automati Fa ial Expression Analysis: A Survey , 1999 .

[55]  J. Russell,et al.  The psychology of facial expression: Frontmatter , 1997 .

[56]  Jeffrey F. Cohn,et al.  Foundations of human computing: facial expression and emotion , 2006, ICMI '06.

[57]  Loïc Kessous,et al.  Multimodal user’s affective state analysis in naturalistic interaction , 2010, Journal on Multimodal User Interfaces.

[58]  Maja Pantic,et al.  Machine analysis of facial behaviour: naturalistic and dynamic behaviour , 2009, Philosophical Transactions of the Royal Society B: Biological Sciences.

[59]  P. Ekman,et al.  What the face reveals : basic and applied studies of spontaneous expression using the facial action coding system (FACS) , 2005 .

[60]  Takeo Kanade,et al.  Multi-PIE , 2008, 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition.

[61]  Jeffrey F. Cohn,et al.  Development of perceptual expertise in emotion recognition , 2009, Cognition.

[62]  Lijun Yin,et al.  Evaluation of spatio-temporal regional features For 3D face analysis , 2009, 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[63]  Lijun Yin,et al.  A high-resolution 3D dynamic facial expression database , 2008, 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition.

[64]  Narendra Ahuja,et al.  Detecting Faces in Images: A Survey , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[65]  J. Cassell,et al.  Embodied conversational agents , 2000 .

[66]  Wade Junek,et al.  Mind Reading: The Interactive Guide to Emotions , 2007 .

[67]  M. Guha.  The Oxford Companion to Emotion and the Affective Sciences , 2010 .

[68]  K. Dautenhahn,et al.  Towards interactive robots in autism therapy: background, motivation and challenges , 2004 .

[69]  Fernando De la Torre,et al.  Facial Expression Analysis , 2011, Visual Analysis of Humans.

[70]  Nicu Sebe,et al.  Facial expression recognition from video sequences: temporal and static modeling , 2003, Comput. Vis. Image Underst..

[71]  John J. B. Allen,et al.  The handbook of emotion elicitation and assessment , 2007 .

[72]  Qiang Ji,et al.  Facial Action Unit Recognition by Exploiting Their Dynamic and Semantic Relationships , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[73]  Narendra Ahuja,et al.  Face Detection and Gesture Recognition for Human-Computer Interaction , 2001, The International Series in Video Computing.

[74]  Rosalind W. Picard.  Affective Computing , 1997 .

[75]  J. Borod.  The Neuropsychology of emotion , 2000 .

[76]  Bülent Sankur,et al.  Robust facial landmarking for registration , 2007, Ann. des Télécommunications.

[77]  Maja Pantic,et al.  Automatic Analysis of Facial Expressions: The State of the Art , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[78]  R. Adolphs.  Recognizing emotion from facial expressions: psychological and neurological mechanisms. , 2002, Behavioral and cognitive neuroscience reviews.

[79]  Y. F. Chong,et al.  Facial expression classification , 2010 .

[80]  Mikkel B. Stegmann,et al.  Analysis and segmentation of face images using point annotations and linear subspace techniques, in , 2002 .

[81]  N. Kanwisher,et al.  The Fusiform Face Area: A Module in Human Extrastriate Cortex Specialized for Face Perception , 1997, The Journal of Neuroscience.

[82]  Zhihong Zeng,et al.  A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[83]  P. Ekman,et al.  Facial action coding system: a technique for the measurement of facial movement , 1978 .

[84]  K. Scherer,et al.  Handbook of affective sciences. , 2003 .

[85]  Daniel Gatica-Perez,et al.  Automatic nonverbal analysis of social interaction in small groups: A review , 2009, Image Vis. Comput..

[86]  Jennifer Healey,et al.  Toward Machine Emotional Intelligence: Analysis of Affective Physiological State , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[87]  Jun Wang,et al.  A 3D facial expression database for facial behavior research , 2006, 7th International Conference on Automatic Face and Gesture Recognition (FGR06).

[88]  J. Russell.  A circumplex model of affect. , 1980 .

[89]  Thomas S. Huang,et al.  3D facial expression recognition based on automatically selected features , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[90]  Takeo Kanade,et al.  Facial Expression Analysis , 2011, AMFG.

[91]  Gwen Littlewort,et al.  Recognizing facial expression: machine learning and application to spontaneous behavior , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[92]  Takeo Kanade,et al.  Recognizing Action Units for Facial Expression Analysis , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[93]  Klaus R. Scherer,et al.  Oxford Companion to Emotion and the Affective Sciences , 2009 .