The MPI Facial Expression Database — A Validated Database of Emotional and Conversational Facial Expressions

The ability to communicate is one of the core aspects of human life. For this, we use not only verbal but also nonverbal signals of remarkable complexity. Among the latter, facial expressions belong to the most important information channels. Despite the large variety of facial expressions we use in daily life, research on facial expressions has so far mostly focused on the emotional aspect. Consequently, most databases of facial expressions available to the research community also include only emotional expressions, neglecting the largely unexplored aspect of conversational expressions. To fill this gap, we present the MPI facial expression database, which contains a large variety of natural emotional and conversational expressions. The database contains 55 different facial expressions performed by 19 German participants. Expressions were elicited with the help of a method-acting protocol, which guarantees both well-defined and natural facial expressions. The method-acting protocol was based on every-day scenarios, which are used to define the necessary context information for each expression. All facial expressions are available in three repetitions, in two intensities, as well as from three different camera angles. A detailed frame annotation is provided, from which a dynamic and a static version of the database have been created. In addition to describing the database in detail, we also present the results of an experiment with two conditions that serve to validate the context scenarios as well as the naturalness and recognizability of the video sequences. Our results provide clear evidence that conversational expressions can be recognized surprisingly well from visual information alone. The MPI facial expression database will enable researchers from different research fields (including the perceptual and cognitive sciences, but also affective computing, as well as computer vision) to investigate the processing of a wider range of natural facial expressions.

[1]  Andrew Sears and Julie A. Jacko.  The human-computer interaction handbook , 2013 .

[2]  Shigeru Akamatsu,et al.  Dynamic Properties Influence the Perception of Facial Expressions , 2001, Perception.

[3]  Christian Peter,et al.  Emotion in Human-Computer Interaction , 2012, Expanding the Frontiers of Visual Analytics and Visualization.

[4]  P. Wilson,et al.  The Nature of Emotions , 2012 .

[5]  A. Calder.  Does Facial Identity and Facial Expression Recognition Involve Separate Visual Routes , 2011 .

[6]  Fadi Dornaika,et al.  Improving dynamic facial expression recognition with feature subset selection , 2011, Pattern Recognit. Lett..

[7]  Tomomi Fujimura,et al.  Effects of Dynamic Information in Recognising Facial Expressions on Dimensional and Categorical Judgments , 2010, Perception.

[8]  Natalie C. Ebner,et al.  FACES—A database of facial expressions in young, middle-aged, and older women and men: Development and validation , 2010, Behavior research methods.

[9]  C. Wallraven,et al.  Dynamic information for the recognition of conversational expressions. , 2009, Journal of vision.

[10]  Rachael E. Jack,et al.  Cultural Confusions Show that Facial Expressions Are Not Universal , 2009, Current Biology.

[11]  M. Herrmann,et al.  Emotions in motion: Dynamic compared to static facial expressions of disgust and happiness reveal more widespread emotion-specific activations , 2009, Brain Research.

[12]  Skyler T. Hawk,et al.  "Worth a thousand words": absolute and relative decoding of nonlinguistic affect vocalizations. , 2009, Emotion.

[13]  Christopher J. Fox,et al.  Defining the face processing network: Optimization of the functional localizer in fMRI , 2009, Human brain mapping.

[14]  J. Schultz,et al.  Natural facial motion enhances cortical responses to faces , 2009, Experimental Brain Research.

[15]  S. McCullough,et al.  Categorical perception of affective and linguistic facial expressions , 2009, Cognition.

[16]  Natalie M. Myres,et al.  Distinctive Paleo-Indian Migration Routes from Beringia Marked by Two Rare mtDNA Haplogroups , 2009, Current Biology.

[17]  Zhihong Zeng,et al.  A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[18]  C. Darwin,et al.  The Expression of the Emotions in Man and Animals , 1956 .

[19]  H. Bülthoff,et al.  The contribution of different facial regions to the recognition of conversational expressions. , 2008, Journal of vision.

[20]  A. Tcherkassof,et al.  Facial expressions of emotions: A methodological contribution to the study of spontaneous and dynamic emotional faces , 2007 .

[21]  Wade Junek,et al.  Mind Reading: The Interactive Guide to Emotions , 2007 .

[22]  S. G. Nikolov,et al.  Turning the other cheek: the viewpoint dependence of facial expression after-effects , 2007, Proceedings of the Royal Society B: Biological Sciences.

[23]  Denis G. Pelli,et al.  ECVP '07 Abstracts , 2007, Perception.

[24]  Hillary Anger Elfenbein,et al.  Toward a dialect theory: cultural differences in the expression and recognition of posed facial expressions. , 2007, Emotion.

[25]  Ines Jentzsch,et al.  Event related potentials and the perception of intensity in facial expressions , 2006, Neuropsychologia.

[26]  J. Cohn,et al.  Movement Differences between Deliberate and Spontaneous Facial Expressions: Zygomaticus Major Action in Smiling , 2006, Journal of nonverbal behavior.

[27]  J. Bavelas,et al.  Nonverbal and Verbal Communication: Hand Gestures and Facial Displays as Part of Language Use in Face-to-face Dialogue. , 2006 .

[28]  Steve Stewart-Williams,et al.  Face recognition and aging: Effects of target age and memory load , 2005, Memory & cognition.

[29]  A. Young,et al.  Understanding the recognition of facial identity and facial expression , 2005, Nature Reviews Neuroscience.

[30]  Heinrich H. Bülthoff,et al.  Manipulating Video Sequences to Determine the Components of Conversational Facial Expressions , 2005, TAP.

[31]  Roddy Cowie,et al.  Beyond emotion archetypes: Databases for emotion modelling using neural networks , 2005, Neural Networks.

[32]  J. Cohn,et al.  Deciphering the Enigmatic Face , 2005, Psychological science.

[33]  J. V. Kuppevelt,et al.  Advances in natural multimodal dialogue systems , 2005 .

[34]  K. Scherer,et al.  Emotions in everyday life: probability of occurrence, risk factors, appraisal and reaction patterns , 2004 .

[35]  Joan Welkowitz,et al.  Changes in Posed Facial Expression of Emotion Across the Adult Life Span , 2004, Experimental aging research.

[36]  Heinrich H. Bülthoff,et al.  View dependence of complex versus simple facial motions , 2004, APGV '04.

[37]  E. Naito,et al.  Enhanced neural activity in response to dynamic facial expressions of emotion: an fMRI study. , 2004, Brain research. Cognitive brain research.

[38]  Nicu Sebe,et al.  The State-of-the-Art in Human-Computer Interaction , 2004, ECCV Workshop on HCI.

[39]  Heinrich H. Bülthoff,et al.  The MPI VideoLab: A system for high quality synchronous recording of video and audio from multiple viewpoints , 2004 .

[40]  Jeffrey F. Cohn,et al.  The Timing of Facial Motion in posed and Spontaneous Smiles , 2003, Int. J. Wavelets Multiresolution Inf. Process..

[41]  Azriel Rosenfeld,et al.  Face recognition: A literature survey , 2003, CSUR.

[42]  G. McCarthy,et al.  Dynamic perception of facial affect and identity in the human brain. , 2003, Cerebral Cortex.

[43]  Clinton D. Kilts,et al.  Dissociable Neural Pathways Are Involved in the Recognition of Emotion in Static and Dynamic Facial Expressions , 2003, NeuroImage.

[44]  S. Baron-Cohen,et al.  Mind Reading: The Interactive Guide to Emotions , 2003 .

[45]  Yajie Tian,et al.  Handbook of face recognition , 2003 .

[46]  K. Scherer,et al.  Handbook of affective sciences. , 2003 .

[47]  Beat Fasel,et al.  Automati Fa ial Expression Analysis: A Survey , 1999 .

[48]  Daniel B Wright,et al.  Age Differences in Lineup Identification Accuracy: People Are Better with Their Own Age , 2002, Law and human behavior.

[49]  Catherine Pelachaud,et al.  Subtleties of facial expressions in embodied agents , 2002, Comput. Animat. Virtual Worlds.

[50]  Matthew Stone,et al.  Making discourse visible: coding and animating conversational facial displays , 2002, Proceedings of Computer Animation 2002 (CA 2002).

[51]  A. O'Toole,et al.  Recognizing moving faces: a psychological and neural synthesis , 2002, Trends in Cognitive Sciences.

[52]  R. Bajcsy,et al.  A method for obtaining 3-dimensional facial expressions and its standardization for use in neurocognitive studies , 2002, Journal of Neuroscience Methods.

[53]  R. Adolphs.  Recognizing emotion from facial expressions: psychological and neurological mechanisms. , 2002, Behavioral and cognitive neuroscience reviews.

[54]  Hillary Anger Elfenbein,et al.  On the universality and cultural specificity of emotion recognition: a meta-analysis. , 2002, Psychological bulletin.

[55]  Hillary Anger Elfenbein,et al.  Is there an in-group advantage in emotion recognition? , 2002, Psychological bulletin.

[56]  J. Cassell,et al.  Nudge nudge wink wink: elements of face-to-face conversation for embodied conversational agents , 2001 .

[57]  J. Cohn,et al.  A Psychometric Evaluation of the Facial Action Coding System for Assessing Spontaneous Expression , 2001 .

[58]  R. Plutchik.  The Nature of Emotions , 2001, American Scientist.

[59]  M. Frank,et al.  The forced-choice paradigm and the perception of facial expressions of emotion. , 2001, Journal of personality and social psychology.

[60]  J. Bavelas,et al.  Listeners as co-narrators. , 2000, Journal of personality and social psychology.

[61]  Maja Pantic,et al.  Automatic Analysis of Facial Expressions: The State of the Art , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[62]  J. Haxby,et al.  The distributed human neural system for face perception , 2000, Trends in Cognitive Sciences.

[63]  Justine Cassell,et al.  Embodied conversational interface agents , 2000, CACM.

[64]  J. Cassell,et al.  Embodied conversational agents , 2000 .

[65]  K. Scherer,et al.  Studying the dynamics of emotional expression using synthesized facial muscle movements. , 2000, Journal of personality and social psychology.

[66]  A J Shinkfield,et al.  Recognition of facial emotional expressions from moving and static displays by individuals with mental retardation. , 1999, American journal of mental retardation : AJMR.

[67]  T. Dalgleish,et al.  Handbook of cognition and emotion , 1999 .

[68]  S. Schweinberger,et al.  Asymmetric relationships among perceptions of facial identity, emotion, and facial speech. , 1998, Journal of experimental psychology. Human perception and performance.

[69]  K. Edwards,et al.  The Face of Time: Temporal Cues in Facial Expressions of Emotion , 1998 .

[70]  U. Hess,et al.  The Intensity of Emotional Facial Expressions and Decoding Accuracy , 1997 .

[71]  S. Demleitner.  [Communication without words]. , 1997, Pflege aktuell.

[72]  J. Russell,et al.  The psychology of facial expression: Frontmatter , 1997 .

[73]  F. Boiten,et al.  Human Facial Expressions , 1996 .

[74]  H. Bülthoff,et al.  Face recognition under varying poses: The role of texture and shape , 1996, Vision Research.

[75]  Simon Baron-Cohen,et al.  Reading the mind in the face: A cross-cultural and developmental study , 1996 .

[76]  G. Rhodes.  Superportraits: Caricatures and Recognition , 1996 .

[77]  P. Ekman,et al.  Strong evidence for universals in facial expressions: a reply to Russell's mistaken critique. , 1994, Psychological bulletin.

[78]  C. Izard.  Innate and universal facial expressions: evidence from developmental and cross-cultural research. , 1994, Psychological bulletin.

[79]  J. Russell.  Is there universal recognition of emotion from facial expression? A review of the cross-cultural studies. , 1994, Psychological bulletin.

[80]  Michael T. Motley,et al.  Facial Affect and Verbal Context in Conversation: Facial Expression as Interjection. , 1993 .

[81]  Daniel Thalmann,et al.  Computer Animation , 1990, Computer Science Workbench.

[82]  A. Ortony,et al.  What's basic about basic emotions? , 1990, Psychological review.

[83]  R. Riggio,et al.  The role of social skills in encoding posed and spontaneous facial expressions , 1988 .

[84]  M. T. Motley,et al.  Facial expression of emotion: A comparison of posed expressions versus spontaneous expressions in an interpersonal communication setting , 1988 .

[85]  P. Shaver,et al.  Emotion knowledge: further exploration of a prototype approach. , 1987, Journal of personality and social psychology.

[86]  A. Young,et al.  Understanding face recognition. , 1986, British journal of psychology.

[87]  A. J. Fridlund,et al.  Emotions and facial expression. , 1985, Science.

[88]  A. Wiens,et al.  Encoding-decoding of nonverbal emotional messages: Sex differences in spontaneous and enacted expressions , 1980 .

[89]  J. N. Bassili.  Facial motion in the perception of faces and of emotional expression. , 1978, Journal of experimental psychology. Human perception and performance.

[90]  P. Ekman,et al.  Facial action coding system: a technique for the measurement of facial movement , 1978 .

[91]  Eleanor Rosch,et al.  Principles of Categorization , 1978 .

[92]  J. R. Landis,et al.  The measurement of observer agreement for categorical data. , 1977, Biometrics.

[93]  R. S. Nickerson,et al.  On conversational interaction with computers , 1976, UODIGS '76.

[94]  D. Barash.  Human Ethology , 1973 .

[95]  R. Birdwhistell.  Kinesics and context , 1970 .

[96]  V. Yngve.  On getting a word in edgewise , 1970 .

[97]  W. A. Hunt.  Recent developments in the field of emotion. , 1941 .