Multimodal Emotion Expressions of Virtual Agents, Mimic and Vocal Emotion Expressions and Their Effects on Emotion Recognition

Emotional expressions of virtual agents are widely believed to enhance the interaction with the user by utilizing more natural means of communication. However, as a result of the current technology virtual agents are often only able to produce facial expressions to convey emotional meaning. The presented research investigates the effects of unimodal vs. multimodal expressions of emotions on the users' recognition of the respective emotional state. We found that multimodal expressions of emotions yield the highest recognition rates. Additionally, emotionally neutral cues in one modality, when presented together with emotionally relevant cues in the other modality, impair the recognition of the correct emotion category as well as intense emotional states.

[1]  K. Scherer,et al.  A blueprint for affective computing: A sourcebook , 2010 .

[2]  Jonathan Gratch,et al.  The Effect of Virtual Agents' Emotion Displays and Appraisals on People's Decision Making in Negotiation , 2012, IVA.

[3]  L. Festinger,et al.  A Theory of Cognitive Dissonance , 2017 .

[4]  Michael Thornton Wyman.  4 – Half-Life 2 , 2011 .

[5]  Jonathan Gratch,et al.  INFLUENCE OF AUTONOMIC SIGNALS ON PERCEPTION OF EMOTIONS IN EMBODIED AGENTS , 2010, Appl. Artif. Intell..

[6]  Burkhardt,et al.  Synthesis of emotional speech , 2010 .

[7]  K. Scherer,et al.  Vocal expression of affect , 2005 .

[8]  D. Hodgson.  Half-Life 2 , 2004 .

[9]  K. Scherer.  Personality inference from voice quality: The loud voice of extroversion. , 1978 .

[10]  Eva G Krumhuber,et al.  FACSGen 2.0 animation software: generating three-dimensional FACS-valid facial expressions for emotion research. , 2012, Emotion.

[11]  Katherine B. Martin,et al.  Facial Action Coding System , 2015 .

[12]  Jenay M. Beer,et al.  Copyright 2009 by Human Factors and Ergonomics Society, Inc. All rights reserved. 10.1518/107118109X12524441079300 , 2009 .

[13]  K. Scherer,et al.  Emotion recognition from expressions in face, voice, and body: the Multimodal Emotion Recognition Test (MERT). , 2009, Emotion.

[14]  Felix D. Schönbrodt,et al.  The Challenge of Constructing Psychologically Believable Agents , 2011, J. Media Psychol. Theor. Methods Appl..

[15]  J. Cassell,et al.  Embodied conversational agents , 2000 .

[16]  J. Vroomen,et al.  The perception of emotions by ear and by eye , 2000 .

[17]  Michael Erb,et al.  Audiovisual integration of emotional signals in voice and face: An event-related fMRI study , 2007, NeuroImage.

[18]  Daniel A. Schneider,et al.  The differential contribution of facial expressions, prosody, and speech content to empathy , 2012, Cognition & emotion.

[19]  Russell Beale,et al.  Psychological responses to simulated displays of mismatched emotional expressions , 2008, Interact. Comput..

[20]  Andrew Ortony,et al.  The Cognitive Structure of Emotions , 1988 .

[21]  Fiorella de Rosis,et al.  Evaluating a realistic agent in an advice-giving task , 2005, Int. J. Hum. Comput. Stud..

[22]  K. Scherer,et al.  Multimodal expression of emotion: affect programs or componential appraisal patterns? , 2007, Emotion.