Automated Detection of Stressful Conversations Using Wearable Physiological and Inertial Sensors

Stressful conversation is a frequently occurring stressor in our daily life. Stressors not only adversely affect our physical and mental health but also our relationships with family, friends, and coworkers. In this paper, we present a model to automatically detect stressful conversations using wearable physiological and inertial sensors. We conducted a lab and a field study with cohabiting couples to collect ecologically valid sensor data with temporally-precise labels of stressors. We introduce the concept of stress cycles, i.e., the physiological arousal and recovery, within a stress event. We identify several novel features from stress cycles and show that they exhibit distinguishing patterns during stressful conversations when compared to physiological response due to other stressors. We observe that hand gestures also show a distinct pattern when stress occurs due to stressful conversations. We train and test our model using field data collected from 38 participants. Our model can determine whether a detected stress event is due to a stressful conversation with an F1-score of 0.83, using features obtained from only one stress cycle, facilitating intervention delivery within 3.9 minutes since the start of a stressful conversation.

[1]  Shrikanth S. Narayanan,et al.  Using Multimodal Wearable Technology to Detect Conflict among Couples , 2017, Computer.

[2]  Tamara Munzner,et al.  Visualization Analysis and Design , 2014, A.K. Peters visualization series.

[3]  J Licinio,et al.  Cardiovascular and neuroendocrine adjustment to public speaking and mental arithmetic stressors. , 1997, Psychophysiology.

[4]  M. Myrtek,et al.  Perception of emotions in everyday life: studies with patients and normals , 1996, Biological Psychology.

[5]  Emre Ertin,et al.  From Markers to Interventions: The Case of Just-in-Time Stress Intervention , 2017, Mobile Health - Sensors, Analytic Methods, and Applications.

[6]  Ross Wilson,et al.  Self-Defining Memory Cues: Creative Expression and Emotional Meaning , 2015, CHI Extended Abstracts.

[7]  E. D. de Geus,et al.  Regular exercise and aerobic fitness in relation to psychological make‐up and physiological stress reactivity. , 1993, Psychosomatic medicine.

[8]  Henry A. Kautz,et al.  Capturing Spontaneous Conversation and Social Dynamics: A Privacy-Sensitive Data Collection Effort , 2007, 2007 IEEE International Conference on Acoustics, Speech and Signal Processing - ICASSP '07.

[9]  Santosh Kumar,et al.  rConverse , 2018, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[10]  Emre Ertin,et al.  mConverse: inferring conversation episodes from respiratory measurements collected in the field , 2011, Wireless Health.

[11]  K. Höök,et al.  Reflecting on the design process of affective health , 2011 .

[12]  Shrikanth S. Narayanan,et al.  Acoustic-Prosodic and Physiological Response to Stressful Interactions in Children with Autism Spectrum Disorder , 2017, INTERSPEECH.

[13]  L. Coleman,et al.  Couple relationships: Why are they important for health and wellbeing? , 2013 .

[14]  Siddharth Shah,et al.  AutoSense: unobtrusively wearable sensor suite for inferring the onset, causality, and consequences of stress in the field , 2011, SenSys.

[15]  Mehrdad Nourani,et al.  A Non-EEG Biosignals Dataset for Assessment and Visualization of Neurological Status , 2016, 2016 IEEE International Workshop on Signal Processing Systems (SiPS).

[16]  N. H. Rod,et al.  High perceived stress and social interaction behaviour among young adults. A study based on objective measures of face-to-face and smartphone interactions , 2019, PloS one.

[17]  Jun Li,et al.  Crowd++: unsupervised speaker count with smartphones , 2013, UbiComp.

[18]  Moushumi Sharmin,et al.  Estimating Drivers' Stress from GPS Traces , 2014, AutomotiveUI.

[19]  Ricardo Gutierrez-Osuna,et al.  Development and Evaluation of an Ambulatory Stress Monitor Based on Wearable Sensors , 2012, IEEE Transactions on Information Technology in Biomedicine.

[20]  A. Muaremi,et al.  Towards Measuring Stress with Smartphones and Wearable Devices During Workday and Sleep , 2013, BioNanoScience.

[21]  Inbal Nahum-Shani,et al.  Finding Significant Stress Episodes in a Discontinuous Time Series of Rapidly Varying Mobile Sensor Data , 2016, CHI.

[22]  R. Andridge,et al.  When couples’ hearts beat together: Synchrony in heart rate variability during conflict predicts heightened inflammation throughout the day , 2018, Psychoneuroendocrinology.

[23]  Kyoungwoo Lee,et al.  Respiration signal based two layer stress recognition across non-verbal and verbal situations , 2020, SAC.

[24]  P. Pietromonaco,et al.  Dating couples' attachment styles and patterns of cortisol reactivity and recovery in response to a relationship conflict. , 2006, Journal of personality and social psychology.

[25]  Inseok Hwang,et al.  SocioPhone: everyday face-to-face interaction monitoring platform using multi-phone sensor fusion , 2013, MobiSys '13.

[26]  R. Kessler,et al.  Everyday stressors and gender differences in daily distress. , 1998, Journal of personality and social psychology.

[27]  Kristof Van Laerhoven,et al.  Wearable affect and stress recognition: A review , 2018, ArXiv.

[28]  Léon J. M. Rothkrantz,et al.  Recognizing Stress Using Semantics and Modulation of Speech and Gestures , 2016, IEEE Transactions on Affective Computing.

[29]  S. Dandapat,et al.  Speaker recognition under stressed condition , 2010, Int. J. Speech Technol..

[30]  Deepak Ganesan,et al.  Hierarchical Active Learning for Model Personalization in the Presence of Label Scarcity , 2019, 2019 IEEE 16th International Conference on Wearable and Implantable Body Sensor Networks (BSN).

[31]  Raul I. Ramos-Garcia,et al.  Using respiratory signals for the recognition of human activities , 2016, 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[32]  Theodora Chaspari,et al.  Population-specific Detection of Couples' Interpersonal Conflict using Multi-task Learning , 2018, ICMI.

[33]  Daniel Gatica-Perez,et al.  StressSense: detecting stress in unconstrained acoustic environments using smartphones , 2012, UbiComp.

[34]  G. Spanier Measuring Dyadic Adjustment: new scales for assessing the quality of marriage and similar dyads , 1976 .

[35]  D. Snyder,et al.  Evidence‐based couple therapy: current status and future directions , 2012 .

[36]  Jennifer Healey,et al.  Detecting stress during real-world driving tasks using physiological sensors , 2005, IEEE Transactions on Intelligent Transportation Systems.

[37]  Carlo Menon,et al.  Detection of Talking in Respiratory Signals: A Feasibility Study Using Machine Learning and Wearable Textile-Based Sensors , 2018, Sensors.

[38]  Daniel Gatica-Perez,et al.  Discovering places of interest in everyday life from smartphone data , 2011, Multimedia Tools and Applications.

[39]  Daniel P. Siewiorek,et al.  Understanding how visual representations of location feeds affect end-user privacy concerns , 2011, UbiComp '11.

[40]  Emre Ertin,et al.  Are we there yet?: feasibility of continuous stress assessment via wireless physiological sensors , 2014, BCB.

[41]  J. Kiecolt-Glaser,et al.  Close relationships, inflammation, and health , 2010, Neuroscience & Biobehavioral Reviews.

[42]  Himanshu Thapliyal,et al.  Validating Physiological Stress Detection Model Using Cortisol as Stress Bio Marker , 2020, 2020 IEEE International Conference on Consumer Electronics (ICCE).

[43]  Inbal Nahum-Shani,et al.  Visualization of time-series sensor data to inform the design of just-in-time adaptive stress interventions , 2015, UbiComp.

[44]  Pierluigi Casale,et al.  Towards Stress Detection in Real-Life Scenarios Using Wearable Sensors: Normalization Factor to Reduce Variability in Stress Physiology , 2016, eHealth 360°.

[45]  Ian M. MacFarlane,et al.  Why is Everyone So Anxious?: An Exploration of Stress and Anxiety in Genetic Counseling Graduate Students , 2011, Journal of Genetic Counseling.

[46]  Emre Ertin,et al.  Continuous inference of psychological stress from sensory measurements collected in the natural environment , 2011, Proceedings of the 10th ACM/IEEE International Conference on Information Processing in Sensor Networks.

[47]  S. Huffel,et al.  Instantaneous changes in heart rate regulation due to mental load in simulated office work , 2011, European Journal of Applied Physiology.

[48]  Elaine Wethington,et al.  The Daily Inventory of Stressful Events , 2002, Assessment.

[49]  Cem Ersoy,et al.  Continuous Stress Detection Using Wearable Sensors in Real Life: Algorithmic Programming Contest Case Study , 2019, Sensors.

[50]  Anne H. Anderson,et al.  The Hcrc Map Task Corpus , 1991 .

[51]  P. Shrout,et al.  Close Relationships and Health in Daily Life: A Review and Empirical Data on Intimacy and Somatic Symptoms , 2012, Psychosomatic medicine.

[52]  B. McEwen,et al.  Allostatic load biomarkers of chronic stress and impact on health and cognition , 2010, Neuroscience & Biobehavioral Reviews.

[53]  Emre Ertin,et al.  cStress: towards a gold standard for continuous stress assessment in the mobile environment , 2015, UbiComp.

[54]  Michal Muszynski,et al.  Can Prediction of Turn-management Willingness Improve Turn-changing Modeling? , 2020, IVA.

[55]  Eva H. Telzer,et al.  A Preliminary Study of Daily Interpersonal Stress and C-Reactive Protein Levels Among Adolescents From Latin American and European Backgrounds , 2009, Psychosomatic medicine.

[56]  Richard W. Bohannon Comfortable and maximum walking speed of adults aged 20-79 years: reference values and determinants. , 1997, Age and ageing.

[57]  D. Kahneman,et al.  A Survey Method for Characterizing Daily Life Experience: The Day Reconstruction Method , 2004, Science.