Auditory-Induced Emotion: A Neglected Channel for Communication in Human-Computer Interaction

Interpreting and responding to affective states of a user is crucial for future intelligent systems. Until recently, the role of sound in affective responses has been frequently ignored. This article provides a brief overview of the research targeting affective reactions to everyday, ecological sounds. This research shows that the subjective interpretation and meaning that listeners attribute to sound, the spatial dimension, or the interactions with other sensory modalities, are as important as the physical properties of sound in evoking an affective response. Situation appraisal and individual differences are also discussed as factors influencing the emotional reactions to auditory stimuli. A study with heartbeat sounds exemplifies some of the introduced ideas and research methodologies, and shows the potential of sound in inducing emotional states.

[1]  K. Scherer,et al.  EMOTIONAL EFFECTS OF MUSIC: PRODUCTION RULES , 2001 .

[2]  Helge J. Ritter,et al.  Sound and meaning in auditory data display , 2004, Proceedings of the IEEE.

[3]  Daniel Västfjäll,et al.  The Subjective Sense of Presence, Emotion Recognition, and Experienced Emotions in Auditory Virtual Environments , 2003, Cyberpsychology Behav. Soc. Netw..

[4]  Daniel Vastfjall.  Influences of current mood and noise sensitivity on judgments of noise annoyance. , 2002, The Journal of psychology.

[5]  Rudolf Bisping.  Emotional Effect of Car Interior Sounds: Pleasantness and Power and Their Relation to Acoustic Key Features , 1995 .

[6]  Christian Martyn Jones,et al.  Acoustic Emotion Recognition for Affective Computer Gaming , 2008, Affect and Emotion in Human-Computer Interaction.

[7]  Nicola Millard,et al.  In the Moodie: Using 'Affective Widgets' to Help Contact Centre Advisors Fight Stress , 2008, Affect and Emotion in Human-Computer Interaction.

[8]  J. E. Creighton,et al.  Lectures On Human And Animal Psychology , 1896 .

[9]  Marc Hassenzahl,et al.  Beyond Task Completion in the Workplace: Execute, Engage, Evolve, Expand , 2008, Affect and Emotion in Human-Computer Interaction.

[10]  Daniel Västfjäll,et al.  Sound Representing Self-Motion in Virtual Environments Enhances Linear Vection , 2008, PRESENCE: Teleoperators and Virtual Environments.

[11]  N. Frijda.  Emotions are functional, most of the time , 1994 .

[12]  Marc Schröder,et al.  Emotional speech synthesis: a review , 2001, INTERSPEECH.

[13]  Russell Beale,et al.  Simulated Emotion in Affective Embodied Agents , 2008, Affect and Emotion in Human-Computer Interaction.

[14]  Stanley B. Woll,et al.  The effects of false feedback on attributed arousal and rated attractiveness in female subjects1 , 1979 .

[15]  Kari Kallinen,et al.  Comparing speakers versus headphones in listening to news from a computer - individual differences and psychophysiological responses , 2004, Comput. Hum. Behav..

[16]  E. N. Solokov.  Perception and the conditioned reflex , 1963 .

[17]  R. Bisping,et al.  Car interior sound quality : experimental analysis by synthesis , 1997 .

[18]  D. Watson,et al.  Toward a consensual structure of mood. , 1985, Psychological bulletin.

[19]  Jörn Loviscach,et al.  In the Mood: Tagging Music with Affects , 2008, Affect and Emotion in Human-Computer Interaction.

[20]  D. Spalding.  The Principles of Psychology , 1873, Nature.

[21]  Leysia Palen,et al.  Don't Get Emotional , 2008, Affect and Emotion in Human-Computer Interaction.

[22]  P. Ekman,et al.  Facial signs of emotional experience. , 1980 .

[23]  R. Levenson.  Emotion and the autonomic nervous system: A prospectus for research on autonomic specificity. , 1988 .

[24]  Daniel Västfjäll,et al.  Self-Representation in Mediated Environments: The Experience of Emotions Modulated by Auditory-Vibrotactile Heartbeat , 2008, Cyberpsychology Behav. Soc. Netw..

[25]  K T Kalveram,et al.  Facial EMG responses to auditory stimuli. , 1996, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[26]  B. Shiv,et al.  Heart and Mind in Conflict: The Interplay of Affect and Cognition in Consumer Decision Making , 1999 .

[27]  N. Todd,et al.  Evidence for a behavioral significance of saccular acoustic sensitivity in humans. , 2001, The Journal of the Acoustical Society of America.

[28]  D H Whalen,et al.  The effects of breath sounds on the perception of synthetic speech. , 1995, The Journal of the Acoustical Society of America.

[29]  P. Lang.  Behavioral treatment and bio-behavioral assessment: computer applications , 1980 .

[30]  C. Spence,et al.  Crossmodal Space and Crossmodal Attention , 2004 .

[31]  P. Lang.  International affective picture system (IAPS) : affective ratings of pictures and instruction manual , 2005 .

[32]  E. A. Bjork.  The perceived quality of natural sounds , 1985 .

[33]  Michael Minge,et al.  Consideration of Multiple Components of Emotions in Human-Technology Interaction , 2008, Affect and Emotion in Human-Computer Interaction.

[34]  Chris Barker,et al.  An Experiment on Public Speaking Anxiety in Response to Three Different Types of Virtual Audience , 2002, Presence: Teleoperators & Virtual Environments.

[35]  M. Bradley,et al.  Affective reactions to acoustic stimuli. , 2000, Psychophysiology.

[36]  Mel Slater,et al.  Body Centred Interaction in Immersive Virtual Environments , 1994 .

[37]  D. Västfjäll,et al.  All emotions are not created equal: Reaching beyond the traditional disputes , 2008 .

[38]  R. Beale,et al.  Multiple and Extended Interactions with Affective Embodied Agents , 2006 .

[39]  Clifford Nass,et al.  The media equation - how people treat computers, television, and new media like real people and places , 1996 .

[40]  Woodrow Barfield,et al.  The Sense of Presence within Auditory Virtual Environments , 1996, Presence: Teleoperators & Virtual Environments.

[41]  A Kjellberg,et al.  Facial Emg Responses to Noise , 1994, Perceptual and motor skills.

[42]  Ellen Poliakoff,et al.  The effect of visual threat on spatial attention to touch , 2007, Cognition.

[43]  Joseph E LeDoux,et al.  Contributions of the Amygdala to Emotion Processing: From Animal Models to Human Behavior , 2005, Neuron.

[44]  P. Bertelson,et al.  Multisensory integration, perception and ecological validity , 2003, Trends in Cognitive Sciences.

[45]  Johannes Wagner,et al.  Automatic Recognition of Emotions from Speech: A Review of the Literature and Recommendations for Practical Realisation , 2008, Affect and Emotion in Human-Computer Interaction.

[46]  Isabelle Peretz,et al.  ception and Performance, 26, 1797–1813. Juslin, PN, & Sloboda, JA (2001). Music and emotion: Theory and re-search. New York: Oxford Univer , 2002 .

[47]  M. Bradley,et al.  Emotion, attention, and the startle reflex. , 1990, Psychological review.

[48]  P. Larsson,et al.  Title: Auditory-induced Presence in Mediated Environments and Related Technology , 2005 .

[49]  C. Osgood,et al.  The Measurement of Meaning , 1958 .

[50]  Bruce E. Wexler,et al.  EEG and EMG responses to emotion-evoking stimuli processed without conscious awareness , 1992, Neuropsychologia.

[51]  J. Hietanen,et al.  Facial electromyographic responses to vocal affect expressions. , 1998, Psychophysiology.

[52]  S. Paradiso.  The Emotional Brain: The Mysterious Underpinnings of Emotional Life , 1998 .

[53]  Christian Martyn Jones,et al.  Affective Human-Robotic Interaction , 2008, Affect and Emotion in Human-Computer Interaction.

[54]  U Dimberg,et al.  Facial electromyography and emotional reactions. , 1990, Psychophysiology.

[55]  Wijnand A. IJsselsteijn,et al.  Effects of Sensory Information and Prior Experience on Direct Subjective Ratings of Presence , 1999, Presence: Teleoperators & Virtual Environments.

[56]  William Sims Bainbridge,et al.  Computational Affective Sociology , 2008, Affect and Emotion in Human-Computer Interaction.

[57]  Costas I. Karageorghis,et al.  The circumplex model of affect: A bi-cultural study , 2007 .

[58]  Rosalind W. Picard,et al.  Evaluating affective interactions: Alternatives to asking what users feel , 2005 .

[59]  Anna-Karin Weivert,et al.  Music and Emotion , 2022 .

[60]  P. Lang.  The emotion probe. Studies of motivation and attention. , 1995, The American psychologist.

[61]  Thomas Jürgensohn,et al.  Comparing Two Emotion Models for Deriving Affective States from Physiological Data , 2008, Affect and Emotion in Human-Computer Interaction.

[62]  J. Singer,et al.  Cognitive, social, and physiological determinants of emotional state. , 1962, Psychological review.

[63]  K. Scherer.  Acoustic Concomitants of Emotional Dimensions: Judging Affect from Synthesized Tone Sequences. , 1972 .