Perception of congruent facial and kinesthetic expressions of emotions

The use of virtual avatars, through facial or gestural expressions, is considered to be a main support for affective communication. Currently, different works have studied the potential of a kinesthetic channel for conveying such information. However, they still have not investigated the complementarity between visual and kinesthetic feedback to effectively convey emotion. This paper studies the relation between some emotional dimensions and the visual and kinesthetic modalities. The experimental results show that subjects used visual and kinesthetic feedbacks to evaluate the pleasure and the arousal dimensions, respectively. We also observed a link between the recognition rate of emotions expressed with the visual modality (resp. kinesthetic modality) and the magnitude of that emotion's pleasure dimension (resp. arousal dimension). These different results should help in the selection of feedbacks according to the features of the investigated emotion.

[1]  Timothy W. Bickmore,et al.  Empathic Touch by Relational Agents , 2010, IEEE Transactions on Affective Computing.

[2]  D. Keltner,et al.  Touch communicates distinct emotions. , 2006, Emotion.

[3]  Jean-claude Martin,et al.  Appraising emotional events during a real-time interactive game , 2009, AFFINE '09.

[4]  J. Russell,et al.  Evidence for a three-factor theory of emotions , 1977 .

[5]  K. Scherer,et al.  Emotion expression in body action and posture. , 2012, Emotion.

[6]  Jeremy N. Bailenson,et al.  Virtual Interpersonal Touch: Expressing and Recognizing Emotions Through Haptic Devices , 2007, Hum. Comput. Interact..

[7]  D. Keltner,et al.  The communication of emotion via touch. , 2009, Emotion.

[8]  Beatrice de Gelder,et al.  Real Faces, Real Emotions: Perceiving Facial Expressions in Naturalistic Contexts of Voices, Bodies, and Scenes , 2011 .

[9]  K. Scherer.  What are emotions? And how can they be measured? , 2005 .

[10]  Mehdi Ammi,et al.  The Impact of Combining Kinesthetic and Facial Expression Displays on Emotion Recognition by Users , 2014, Int. J. Hum. Comput. Interact..

[11]  Mehdi Ammi,et al.  Improvement of the recognition of facial expressions with haptic feedback , 2011, 2011 IEEE International Workshop on Haptic Audio Visual Environments and Games.

[12]  S. Baron-Cohen,et al.  The Cambridge Mindreading (CAM) Face-Voice Battery: Testing Complex Emotion Recognition in Adults with and without Asperger Syndrome , 2006, Journal of autism and developmental disorders.

[13]  Mehdi Ammi,et al.  Comparison of statistical methods for analysis of affective haptic expressions , 2012, SAP.

[14]  Johan Wessberg,et al.  Unmyelinated tactile afferents have opposite effects on insular and somatosensory cortical processing , 2008, Neuroscience Letters.

[15]  Adriana Tapus,et al.  Haptic Human-Robot Affective Interaction in a Handshaking Social Protocol , 2015, 2015 10th ACM/IEEE International Conference on Human-Robot Interaction (HRI).