Affective Communication through Air Jet Stimulation: Evidence from Event-Related Potentials

ABSTRACT In this article, effectiveness of air jet stimulation in mediated emotional communication was investigated by assessing cross-modal influences of visual emotional expressions on tactile perception. Brain responses to combined visual faces and air jet stimuli were measured using event-related potentials; whereas, emotional responses were assessed using self-reported pleasantness of the tactile stimulation. ERP results reveal significant differences between the different facial expressions for the same tactile air-jet intensity in the somatosensory area. Moreover, participants’ pleasantness ratings suggest an effect of the visual stimulus on the difference tactile conditions that correspond to air jet stimulation intensities: low, medium, and high. These promising results provide evidence in the potential efficiency of this stimulation technique in activating skin receptors that play an important role in social and affective behaviors.

[1]  Roope Raisamo,et al.  The cutaneous-rabbit illusion: What if it is not a Rabbit? , 2017, 2017 IEEE World Haptics Conference (WHC).

[2]  A. Goldman,et al.  Simulationist models of face-based emotion recognition , 2005, Cognition.

[3]  J. Wessberg,et al.  Human C-Tactile Afferents Are Tuned to the Temperature of a Skin-Stroking Caress , 2014, The Journal of Neuroscience.

[4]  Zhigang Deng,et al.  Multimodal joint information processing in human machine interaction: recent advances , 2014, Multimedia Tools and Applications.

[5]  T. Allison,et al.  Social perception from visual cues: role of the STS region , 2000, Trends in Cognitive Sciences.

[6]  Albino Nogueiras,et al.  Speech emotion recognition using hidden Markov models , 2001, INTERSPEECH.

[7]  D. Keltner,et al.  The communication of emotion via touch. , 2009, Emotion.

[8]  Veikko Surakka,et al.  The effects of affective interventions in human-computer interaction , 2004, Interact. Comput..

[9]  L. Rothkrantz,et al.  Toward an affect-sensitive multimodal human-computer interaction , 2003, Proc. IEEE.

[10]  M. Knapp,et al.  Nonverbal communication in human interaction , 1972 .

[11]  Zhigang Deng,et al.  Analysis of emotion recognition using facial expressions, speech and multimodal information , 2004, ICMI '04.

[12]  Johan Wessberg,et al.  In touch with your emotions: Oxytocin and touch change social impressions while others’ facial expressions can alter touch , 2014, Psychoneuroendocrinology.

[13]  R. Nagarajan,et al.  Real-time system for facial emotion detection using GPSO algorithm , 2010, 2010 IEEE Symposium on Industrial Electronics and Applications (ISIEA).

[14]  Ralph Adolphs,et al.  The neuropsychology of face perception: beyond simple dissociations and functional selectivity , 2011, Philosophical Transactions of the Royal Society B: Biological Sciences.

[15]  Michiel M. A. Spapé,et al.  Feeling Touched: Emotional Modulation of Somatosensory Potentials to Interpersonal Touch , 2017, Scientific Reports.

[16]  J. Wessberg,et al.  Ultra-late EEG potential evoked by preferential activation of unmyelinated tactile afferents in human hairy skin , 2013, Neuroscience Letters.

[17]  Aijun Li,et al.  Prosody conversion from neutral speech to emotional speech , 2006, IEEE Transactions on Audio, Speech, and Language Processing.

[18]  J. G. Taylor,et al.  Emotion recognition in human-computer interaction , 2005, Neural Networks.

[19]  N. Henley.  Status and sex: Some touching observations , 1973 .

[20]  Adriana Tapus,et al.  Haptic Human-Robot Affective Interaction in a Handshaking Social Protocol , 2015, 2015 10th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[21]  Fakhri Karray,et al.  Survey on speech emotion recognition: Features, classification schemes, and databases , 2011, Pattern Recognit..

[22]  Jelle Saldien,et al.  On the Design of the Huggable Robot Probo , 2008 .

[23]  J. Haxby,et al.  Distinct representations of eye gaze and identity in the distributed human neural system for face perception , 2000, Nature Neuroscience.

[24]  Catherine Pelachaud,et al.  Greta: A Simple Facial Animation Engine , 2002 .

[25]  Zheng Li,et al.  Modeling of Layered Fuzzy Facial Expression Generation , 2007, HCI.

[26]  Dirk Heylen,et al.  Generating expressive speech for storytelling applications , 2006, IEEE Transactions on Audio, Speech, and Language Processing.

[27]  Wendy E. Mackay,et al.  CHI '13 Extended Abstracts on Human Factors in Computing Systems , 2013, CHI 2013.

[28]  Shuo Wang,et al.  Squeeze me, but don't tease me: Human and mechanical touch enhance visual attention and emotion discrimination , 2011, Social neuroscience.

[29]  Johan Wessberg,et al.  Discriminative touch and emotional touch. , 2007, Canadian journal of experimental psychology = Revue canadienne de psychologie experimentale.

[30]  Kostas Karpouzis,et al.  Emotion Analysis in Man-Machine Interaction Systems , 2004, MLMI.

[31]  Mehdi Ammi,et al.  EmotionAir: Perception of Emotions from Air Jet Based Tactile Stimulation , 2013, 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction.

[32]  Mehdi Ammi,et al.  The Impact of Combining Kinesthetic and Facial Expression Displays on Emotion Recognition by Users , 2014, Int. J. Hum. Comput. Interact..

[33]  Yuxiao Hu,et al.  Audio-Visual Spontaneous Emotion Recognition , 2007, Artifical Intelligence for Human Computing.

[34]  Mehdi Ammi,et al.  Haptic communication of dimensions of emotions using air jet based tactile stimulation , 2015, Journal on Multimodal User Interfaces.

[35]  Youngwoo Park,et al.  The roles of touch during phone conversations: long-distance couples' use of POKE in their homes , 2013, CHI.

[36]  D. Lundqvist,et al.  Facial expressions of emotion (KDEF): Identification under different display-duration conditions , 2008, Behavior research methods.

[37]  Steve Yohanan,et al.  The Haptic Creature : social human-robot interaction through affective touch , 2012 .

[38]  M. Seghier,et al.  A network of occipito-temporal face-sensitive areas besides the right middle fusiform gyrus is necessary for normal face processing. , 2003, Brain : a journal of neurology.

[39]  Michael Andres,et al.  Dissociable roles of the human somatosensory and superior temporal cortices for processing social face signals , 2004, The European journal of neuroscience.

[40]  G. Essick,et al.  Quantitative assessment of pleasant touch , 2010, Neuroscience & Biobehavioral Reviews.

[41]  E. Vesterinen,et al.  Affective Computing , 2009, Encyclopedia of Biometrics.

[42]  Dzmitry Tsetserukou,et al.  World's first wearable humanoid robot that augments our emotions , 2010, AH.

[43]  M. Eimer,et al.  Event-related brain potential correlates of emotional face processing , 2007, Neuropsychologia.

[44]  M. Borchert,et al.  Emotions in speech - experiments with prosody and quality features in speech for use in categorical and dimensional emotion recognition environments , 2005, 2005 International Conference on Natural Language Processing and Knowledge Engineering.

[45]  John F. Deethardt,et al.  Tactile communication and personality differences , 1983 .

[46]  FragopanagosN.,et al.  2005 Special Issue , 2005 .

[47]  P. Ekman,et al.  Unmasking the face : a guide to recognizing emotions from facial clues , 1975 .

[48]  Mohammad Soleymani,et al.  Best of affective computing and intelligent interaction 2013 in multimodal interactions , 2014, Journal on Multimodal User Interfaces.

[49]  R. Beale,et al.  Multiple and Extended Interactions with Affective Embodied Agents , 2006 .

[50]  G. Essick,et al.  Psychophysical assessment of the affective components of non-painful touch. , 1999, Neuroreport.

[51]  D. Keltner,et al.  Touch communicates distinct emotions. , 2006, Emotion.

[52]  Sonja Utz,et al.  Mediated interpersonal communication in multiplayer video games: Implications for entertainment and relationship management , 2008 .

[53]  Alejandra Sel,et al.  The Emotional Homunculus: ERP Evidence for Independent Somatosensory Responses during Facial Emotional Processing , 2014, The Journal of Neuroscience.

[54]  Panayiotis G. Georgiou,et al.  Real-time Emotion Detection System using Speech: Multi-modal Fusion of Different Timescale Features , 2007, 2007 IEEE 9th Workshop on Multimedia Signal Processing.

[55]  Margot J. Taylor,et al.  Early processing of the six basic facial emotional expressions. , 2003, Brain research. Cognitive brain research.

[56]  L. Mayes,et al.  Face-related ERPs are modulated by point of gaze , 2010, Neuropsychologia.

[57]  J. Haxby,et al.  The distributed human neural system for face perception , 2000, Trends in Cognitive Sciences.

[58]  Radoslaw Niewiadomski,et al.  Model of Facial Expressions Management for an Embodied Conversational Agent , 2007, ACII.

[59]  Johannes Wagner,et al.  Automatic Recognition of Emotions from Speech: A Review of the Literature and Recommendations for Practical Realisation , 2008, Affect and Emotion in Human-Computer Interaction.

[60]  J. Wessberg,et al.  Coding of pleasant touch by unmyelinated afferents in humans , 2009, Nature Neuroscience.

[61]  P. Ekman,et al.  Constants across cultures in the face and emotion. , 1971, Journal of personality and social psychology.

[62]  Munekazu Date,et al.  MulDiRoH: An Evaluation of Facial Direction Expression in Teleconferencing on a Multi-view Display System , 2014, HCI.

[63]  S. Srivastava,et al.  The Big Five Trait taxonomy: History, measurement, and theoretical perspectives. , 1999 .

[64]  Johan Wessberg,et al.  Receptive field properties of unmyelinated tactile afferents in the human skin. , 2003, Journal of neurophysiology.

[65]  Matthew J. Hertenstein,et al.  The Handbook of Touch: Neuroscience, Behavioral, and Health Perspectives , 2011 .

[66]  Jeremy N. Bailenson,et al.  Virtual Interpersonal Touch: Expressing and Recognizing Emotions Through Haptic Devices , 2007, Hum. Comput. Interact..

[67]  Elizabeth Hussey,et al.  Perception of Facial Expression in Somatosensory Cortex Supports Simulationist Models , 2009, The Journal of Neuroscience.

[68]  J. Vroomen,et al.  The perception of emotions by ear and by eye , 2000 .

[69]  Hatice Gunes,et al.  Face and Body Gesture Recognition for a Vision-Based Multimodal Analyzer , 2004, VIP.

[70]  Matthew J. Hertenstein,et al.  The Communicative Functions of Touch in Humans, Nonhuman Primates, and Rats: A Review and Synthesis of the Empirical Research , 2006, Genetic, social, and general psychology monographs.

[71]  Colin Grubb.  Multimodal Emotion Recognition , 2013 .

[72]  Anna Esposito,et al.  Affect in Multimodal Information , 2009, Affective Information Processing.

[73]  George N. Votsis,et al.  Emotion recognition in human-computer interaction , 2001, IEEE Signal Process. Mag..

[74]  Bernd Kleinjohann,et al.  Real-Time Facial Expression Recognition Using a Fuzzy Emotion Model , 2007, 2007 IEEE International Fuzzy Systems Conference.

[75]  Loïc Kessous,et al.  Multimodal emotion recognition from expressive faces, body gestures and speech , 2007, AIAI.

[76]  R. Dolan,et al.  Emotion, Cognition, and Behavior , 2002, Science.

[77]  D. Keltner,et al.  Emotion, Social Function, and Psychopathology , 1998 .

[78]  David Matsumoto,et al.  More evidence for the universality of a contempt expression , 1992 .

[79]  Matthew J. Hertenstein,et al.  Nonverbal channel use in communication of emotion: how may depend on why. , 2011, Emotion.

[80]  Pedro Montoya,et al.  Affective modulation of somatosensory-evoked potentials elicited by tactile stimulation , 2006, Brain Research.

[81]  Céline Clavel,et al.  MARC: a framework that features emotion models for facial animation during human–computer interaction , 2013, Journal on Multimodal User Interfaces.

[82]  Paul Issartel,et al.  HAIR: HAptic feedback with a mobile AIR jet , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[83]  Pierre-Yves Oudeyer,et al.  The production and recognition of emotions in speech: features and algorithms , 2003, Int. J. Hum. Comput. Stud..

[84]  J. Wessberg,et al.  The neurophysiology of unmyelinated tactile afferents , 2010, Neuroscience & Biobehavioral Reviews.

[85]  S. Planalp,et al.  Communication, cognition, and emotion , 1993 .

[86]  Loïc Kessous,et al.  Modeling Naturalistic Affective States Via Facial, Vocal, and Bodily Expressions Recognition , 2007, Artifical Intelligence for Human Computing.

[87]  Danielle Z. Bolling,et al.  Brain mechanisms for processing affective touch , 2013, Human brain mapping.

[88]  Christian Keysers,et al.  Somatosensation in social perception , 2010, Nature Reviews Neuroscience.

[89]  R. Adolphs,et al.  A Role for Somatosensory Cortices in the Visual Recognition of Emotion as Revealed by Three-Dimensional Lesion Mapping , 2000, The Journal of Neuroscience.

[90]  Yuan Gao,et al.  What Does Touch Tell Us about Emotions in Touchscreen-Based Gameplay? , 2012, TCHI.

[91]  D. Grandjean,et al.  Electrophysiological correlates of rapid spatial orienting towards fearful faces. , 2004, Cerebral cortex.

[92]  Aduén Darriba Frederiks,et al.  Towards tactile expressions of emotion through mediated touch , 2013, CHI Extended Abstracts.

[93]  Michiel M. A. Spapé,et al.  Effects of touch on emotional face processing: A study of event-related potentials, facial EMG and cardiac activity , 2017, Biological Psychology.

[94]  Joseph P. McCleery,et al.  Neural time-course of the observation of human and non-human object touch. , 2014, Social cognitive and affective neuroscience.

[95]  Björn W. Schuller,et al.  Context-sensitive multimodal emotion recognition from speech and facial expression using bidirectional LSTM modeling , 2010, INTERSPEECH.

[96]  Cynthia Breazeal,et al.  Affective Touch for Robotic Companions , 2005, ACII.

[97]  R. Adolphs,et al.  Dissociable neural systems for recognizing emotions , 2003, Brain and Cognition.