ThermalWear: Exploring Wearable On-chest Thermal Displays to Augment Voice Messages with Affect

Voice is a rich modality for conveying emotions, however emotional prosody production can be situationally or medically impaired. Since thermal displays have been shown to evoke emotions, we explore how thermal stimulation can augment perception of neutrally-spoken voice messages with affect. We designed ThermalWear, a wearable on-chest thermal display, then tested in a controlled study (N=12) the effects of fabric, thermal intensity, and direction of change. Thereafter, we synthesized 12 neutrally-spoken voice messages, validated (N=7) them, then tested (N=12) if thermal stimuli can augment their perception with affect. We found warm and cool stimuli (a) can be perceived on the chest, and quickly without fabric (4.7-5s) (b) do not incur discomfort (c) generally increase arousal of voice messages and (d) increase / decrease message valence, respectively. We discuss how thermal displays can augment voice perception, which can enhance voice assistants and support individuals with emotional prosody impairments.

[1]  J. Bargh,et al.  Experiencing Physical Warmth Promotes Interpersonal Warmth , 2008, Science.

[2]  Martin Halvey,et al.  Some like it hot: thermal feedback for mobile devices , 2011, CHI.

[3]  Jody Kreiman,et al.  Foundations of Voice Studies: An Interdisciplinary Approach to Voice Production and Perception , 2011 .

[4]  George Havenith,et al.  Regional distribution of thermal sensitivity to cold at rest and during mild exercise in males , 2012 .

[5]  F. Gosselin,et al.  The Montreal Affective Voices: A validated set of nonverbal affect bursts for research on auditory affective processing , 2008, Behavior research methods.

[6]  Wonjun Lee,et al.  Thermo-message: exploring the potential of heat as a modality of peripheral expression , 2010, CHI Extended Abstracts.

[7]  Lynette A. Jones,et al.  Warm or Cool, Large or Small? The Challenge of Thermal Displays , 2008, IEEE Transactions on Haptics.

[8]  Tek-Jin Nam,et al.  Thermal Interaction with a Voice-based Intelligent Agent , 2018, CHI Extended Abstracts.

[9]  Martin Halvey,et al.  The effect of clothing on thermal feedback perception , 2011, ICMI '11.

[10]  Stephen A. Brewster,et al.  Hot Under the Collar: Mapping Thermal Feedback to Dimensional Models of Emotion , 2016, CHI.

[11]  Silke Paulmann,et al.  How Psychological Stress Affects Emotional Prosody , 2016, PloS one.

[12]  Wolfgang Grodd,et al.  Cerebral processing of emotional prosody—influence of acoustic parameters and arousal , 2008, NeuroImage.

[13]  J. Hietanen,et al.  Bodily maps of emotions , 2013, Proceedings of the National Academy of Sciences.

[14]  Chris Berka,et al.  Somneo: Device with Thermal Stimulation for Modulating Sleep Architecture and Enhancing Neuro-Cognitive Function , 2014, HCI.

[15]  Ted Ruffman,et al.  Aging and the Perception of Emotion: Processing Vocal Expressions Alone and With Faces , 2009, Experimental aging research.

[16]  Patricia J. Brooks,et al.  Attention to emotional tone of voice in speech perception in children with autism , 2013 .

[17]  Hauke R. Heekeren,et al.  Emotional prosody processing in autism spectrum disorder , 2016, Social cognitive and affective neuroscience.

[18]  E. Rodero.  Intonation and emotion: influence of pitch levels and contour type on creating emotions. , 2011, Journal of voice : official journal of the Voice Foundation.

[19]  Enrico Glerean,et al.  Bodily maps of emotions are culturally universal. , 2020, Emotion.

[20]  Roope Raisamo,et al.  Emotional responses to thermal stimuli , 2011, ICMI '11.

[21]  Hyun Wook Park,et al.  BRAIN ACTIVATION RELATED TO AFFECTIVE DIMENSION DURING THERMAL STIMULATION IN HUMANS: A FUNCTIONAL MAGNETIC RESONANCE IMAGING STUDY , 2007, The International journal of neuroscience.

[22]  Kouta Minamizawa,et al.  ThermalBracelet: Exploring Thermal Haptic Feedback Around the Wrist , 2019, CHI.

[23]  Inge-Marie Eigsti,et al.  The neural underpinnings of prosody in autism , 2012, Child neuropsychology : a journal on normal and abnormal development in childhood and adolescence.

[24]  Sebastian Müller,et al.  Haptically augmented remote speech communication: a study of user practices and experiences , 2012, NordiCHI.

[25]  Andreas Butz,et al.  Reproducing materials of virtual elements on touchscreens using supplemental thermal feedback , 2012, ICMI '12.

[26]  Haiko Luepsen,et al.  The aligned rank transform and discrete variables: A warning , 2017, Commun. Stat. Simul. Comput..

[27]  Martin Halvey,et al.  "Baby it's cold outside": the influence of ambient temperature and humidity on thermal feedback , 2012, CHI.

[28]  Hugo D. Critchley,et al.  You Turn Me Cold: Evidence for Temperature Contagion , 2014, PloS one.

[29]  D. Cicchetti.  Guidelines, Criteria, and Rules of Thumb for Evaluating Normed and Standardized Assessment Instruments in Psychology. , 1994 .

[30]  P. Niedenthal.  Embodying Emotion , 2007, Science.

[31]  J. J. Higgins,et al.  The aligned rank transform for nonparametric factorial analyses using only anova procedures , 2011, CHI.

[32]  G. Arundale.  The Psychology of the Emotions , 1898, Nature.

[33]  Lynne Baillie,et al.  The Effect of Thermal Stimuli on the Emotional Perception of Images , 2016, CHI.

[34]  Jordan Tewell,et al.  The Heat is On: A Temperature Display for Conveying Affective Feedback , 2017, CHI.

[35]  Norihiro Sadato,et al.  Brain activation during whole body cooling in humans studied with functional magnetic resonance imaging , 2002, Neuroscience Letters.

[36]  Kathleen Peterson.  Six Modifications Of The Aligned Rank Transform Test For Interaction , 2002 .

[37]  Martin Halvey,et al.  Augmenting Media with Thermal Stimulation , 2012, HAID.

[38]  Suguru Higashino,et al.  "Hiya-Atsu" media: augmenting digital media with temperature , 2009, CHI Extended Abstracts.

[39]  Daniel Lundqvist,et al.  The EU-Emotion Voice Database , 2018, Behavior Research Methods.

[40]  Enrico Glerean,et al.  Bodily maps of emotions across child development. , 2016, Developmental science.

[41]  Chris Baber,et al.  The comfort assessment of wearable computers , 2002, Proceedings. Sixth International Symposium on Wearable Computers,.

[42]  Jody Kreiman,et al.  In the Beginning Was the Familiar Voice: Personally Familiar Voices in the Evolutionary and Contemporary Biology of Communication , 2012, Integrative psychological & behavioral science.

[43]  M. Knapp,et al.  Nonverbal communication in human interaction , 1972 .

[44]  Emiliana R. Simon-Thomas,et al.  The voice conveys specific emotions: evidence from vocal burst displays. , 2009, Emotion.

[45]  S. Scott,et al.  Perceptual Cues in Nonverbal Vocal Expressions of Emotion , 2010 .

[46]  Klaus R. Scherer,et al.  Vocal communication of emotion: A review of research paradigms , 2003, Speech Commun..

[47]  George Havenith,et al.  Distribution of skin thermal sensitivity , 2017 .

[48]  Heiga Zen,et al.  WaveNet: A Generative Model for Raw Audio , 2016, SSW.

[49]  Abhijit Majumdar,et al.  Modelling of thermal conductivity of knitted fabrics made of cotton–bamboo yarns using artificial neural network , 2011 .

[50]  Antonio Krüger,et al.  ambiPad: enriching mobile digital media with ambient feedback , 2014, MobileHCI '14.

[51]  M. Abitha,et al.  Thermal Conductivity of Fabrics , 2015 .

[52]  Shannon H Pennington.  Blood, sweat and fears. , 2011, JEMS : a journal of emergency medical services.

[53]  Kevin A Hallgren,et al.  Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial. , 2012, Tutorials in quantitative methods for psychology.

[54]  F. Lepore,et al.  Women process multisensory emotion expressions more efficiently than men , 2010, Neuropsychologia.

[55]  Martin Halvey,et al.  Thermal icons: evaluating structured thermal feedback for mobile interaction , 2012, Mobile HCI.

[56]  K. McGraw,et al.  Forming inferences about some intraclass correlation coefficients. , 1996 .

[57]  D. Keltner,et al.  The communication of emotion via touch. , 2009, Emotion.

[58]  Roope Raisamo,et al.  Cold or Hot? How Thermal Stimuli Are Related to Human Emotional System? , 2013, HAID.

[59]  Dimitri Van De Ville,et al.  Decoding of Emotional Information in Voice-Sensitive Cortices , 2009, Current Biology.

[60]  L. Watts,et al.  Communicating Social Presence Through Thermal Hugs , 2010 .

[61]  Ivan Poupyrev,et al.  "I don't Want to Wear a Screen": Probing Perceptions of and Possibilities for Dynamic Displays on Clothing , 2016, CHI.

[62]  P. Gorelick,et al.  The aprosodias: further functional-anatomical evidence for the organisation of affective language in the right hemisphere. , 1987, Journal of neurology, neurosurgery, and psychiatry.

[63]  Clint Zeagler,et al.  Where to wear it: functional, technical, and social considerations in on-body location for wearable technology 20 years of designing for wearability , 2017, SEMWEB.

[64]  Wonjun Lee,et al.  Explorative research on the heat as an expression medium: focused on interpersonal communication , 2012, Personal and Ubiquitous Computing.

[65]  M. Bradley,et al.  Measuring emotion: the Self-Assessment Manikin and the Semantic Differential. , 1994, Journal of behavior therapy and experimental psychiatry.

[66]  Hans Ijzerman,et al.  The Thermometer of Social Relations , 2009, Psychological science.

[67]  Jun Rekimoto,et al.  AffectPhone: A Handset Device to Present User's Emotional State with Warmth/Coolness , 2016, B-Interface.

[68]  Mary Ritchie Key,et al.  The Relationship of Verbal and Nonverbal Communication , 1980 .

[69]  Stephen A. Brewster,et al.  In the Heat of the Moment: Subjective Interpretations of Thermal Feedback During Interaction , 2015, CHI.

[70]  D. Wildgruber,et al.  Emotional voices in context: a neurobiological model of multimodal affective information processing. , 2011, Physics of life reviews.

[71]  D. Lancker,et al.  Recognition of emotional‐prosodic meanings in speech by autistic, schizophrenic, and normal children , 1989 .

[72]  G. Bryant.  Prosodic Contrasts in Ironic Speech , 2010 .

[73]  Chris Baber,et al.  A Tool to Assess the Comfort of Wearable Computers , 2005, Hum. Factors.

[74]  Helen Tager-Flusberg,et al.  Emotional Facial and Vocal Expressions During Story Retelling by Children and Adolescents With High-Functioning Autism , 2022 .

[75]  Meredith Ringel Morris,et al.  SWARM: An Actuated Wearable for Mediating Affect , 2015, TEI.

[76]  Klaus R. Scherer,et al.  Vocal communication of emotion , 2000 .