Combining Facial and Postural Expressions of Emotions in a Virtual Character

Psychology suggests highly synchronized expressions of emotion across different modalities. Few experiments jointly studied the relative contribution of facial expression and body posture to the overall perception of emotion. Computational models for expressive virtual characters have to consider how such combinations will be perceived by users. This paper reports on two studies exploring how subjects perceived a virtual agent. The first study evaluates the contribution of the facial and postural expressions to the overall perception of basic emotion categories, as well as the valence and activation dimensions. The second study explores the impact of incongruent expressions on the perception of superposed emotions which are known to be frequent in everyday life. Our results suggest that the congruence of facial and bodily expression facilitates the recognition of emotion categories. Yet, judgments were mainly based on the emotion expressed in the face but were nevertheless affected by postures for the perception of the activation dimension.

[1]  P. Ekman.  Unmasking The Face , 1975 .

[2]  Catherine Pelachaud,et al.  Multimodal expressive embodied conversational agents , 2005, ACM Multimedia.

[3]  Hans-Peter Seidel,et al.  Mixed feelings: expression of non-basic emotions in a muscle-based talking head , 2005, Virtual Reality.

[4]  K. Scherer,et al.  The New Handbook of Methods in Nonverbal Behavior Research , 2008 .

[5]  H. Tajfel.  The Social Dimension: European Developments In Social Psychology , 1984 .

[6]  K. Scherer,et al.  Multimodal expression of emotion: affect programs or componential appraisal patterns? , 2007, Emotion.

[7]  J. Harrigan,et al.  Proxemics, kinesics, and gaze. , 2005 .

[8]  D. Ballin,et al.  A framework for interpersonal attitude and non-verbal communication in improvisational visual media production , 2004 .

[9]  Radoslaw Niewiadomski,et al.  Perception of Blended Emotions: From Video Corpus to Expressive Agent , 2006, IVA.

[10]  J. Russell,et al.  Evidence for a three-factor theory of emotions , 1977 .

[11]  H. Meeren,et al.  Rapid perceptual integration of facial expression and emotional body language. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[12]  Andrea Kleinsmith,et al.  Cross-cultural differences in recognizing affect from body posture , 2006, Interact. Comput..

[13]  K. Scherer,et al.  Are facial expressions of emotion produced by categorical affect programs or dynamically driven by appraisal? , 2007, Emotion.

[14]  Judith A. Hall,et al.  Gender differences in judgments of multiple emotions from facial expressions. , 2004, Emotion.

[15]  Hatice Gunes,et al.  Bi-modal emotion recognition from expressive face and body gestures , 2007, J. Netw. Comput. Appl..

[16]  C. W. Hughes.  Emotion: Theory, Research and Experience , 1982 .

[17]  Andrea Kleinsmith,et al.  Recognizing Affective Dimensions from Body Posture , 2007, ACII.

[18]  R. Plutchik.  A GENERAL PSYCHOEVOLUTIONARY THEORY OF EMOTION , 1980 .

[19]  J. Hietanen,et al.  Judgment of other people's facial expressions of emotions is influenced by their concurrent affective hand movements. , 2008, Scandinavian journal of psychology.

[20]  Woontack Woo,et al.  Emotion Analysis from Dance Performance Using Time-Delay Neural Networks , 2000 .

[21]  M. Katz,et al.  The Psychology of depression : contemporary theory and research , 1974 .

[22]  K. Scherer,et al.  Multimodal expression of emotion , 2009 .

[23]  H. Wallbott.  Bodily expression of emotion , 1998 .

[24]  Rosalind W. Picard,et al.  Automated Posture Analysis for Detecting Learner's Interest Level , 2003, 2003 Conference on Computer Vision and Pattern Recognition Workshop.

[25]  K. Scherer,et al.  The World of Emotions is not Two-Dimensional , 2007, Psychological science.

[26]  Nadia Bianchi-Berthouze,et al.  Modeling human affective postures: an information theoretic characterization of posture features: Research Articles , 2004 .

[27]  J. Turner.  Human Emotions: A Sociological Theory , 2007 .

[28]  P. Gosselin,et al.  Le Decodage De Prototypes Emotionnels Faciaux , 1995 .

[29]  P. Ekman,et al.  Handbook of methods in nonverbal behavior research , 1982 .

[30]  Andrea Kleinsmith,et al.  A categorical approach to affective gesture recognition , 2003, Connect. Sci..

[31]  P. Ekman,et al.  Unmasking the face : a guide to recognizing emotions from facial clues , 1975 .

[32]  B. Stuart,et al.  Nonverbal Behavior and Psychopathology , 2008 .

[33]  J. Russell.  A circumplex model of affect. , 1980 .

[34]  Klaus R. Scherer,et al.  Using Actor Portrayals to Systematically Study Multimodal Emotion Expression: The GEMEP Corpus , 2007, ACII.

[35]  P. Ekman,et al.  A new pan-cultural facial expression of emotion , 1986 .

[36]  W. Stroebe,et al.  Introduction to social psychology: A European perspective , 1988 .

[37]  P. Ekman.  An argument for basic emotions , 1992 .

[38]  A. W. Siegman,et al.  Nonverbal behavior and communication , 1979 .