PASS: A Multimodal Database of Physical Activity and Stress for Mobile Passive Body/ Brain-Computer Interface Research

With the burgeoning of wearable devices and passive body/brain-computer interfaces (B/BCIs), automated stress monitoring in everyday settings has gained significant attention recently, with applications ranging from serious games to clinical monitoring. With mobile users, however, challenges arise due to other overlapping (and potentially confounding) physiological responses (e.g., due to physical activity) that may mask the effects of stress, as well as movement artifacts that can be introduced in the measured signals. For example, the classical increase in heart rate can no longer be attributed solely to stress and could be caused by the activity itself. This makes the development of mobile passive B/BCIs challenging. In this paper, we introduce PASS, a multimodal database of Physical Activity and StresS collected from 48 participants. Participants performed tasks of varying stress levels at three different activity levels and provided quantitative ratings of their perceived stress and fatigue levels. To manipulate stress, two video games (i.e., a calm exploration game and a survival game) were used. Peripheral physical activity (electrocardiography, electrodermal activity, breathing, skin temperature) as well as cerebral activity (electroencephalography) were measured throughout the experiment. A complete description of the experimental protocol is provided and preliminary analyses are performed to investigate the physiological reactions to stress in the presence of physical activity. The PASS database, including raw data and subjective ratings has been made available to the research community at http://musaelab.ca/pass-database/. It is hoped that this database will help advance mobile passive B/BCIs for use in everyday settings.

[1]  Tiago H. Falk,et al.  Fusion of Spectral and Spectro-Temporal EEG Features for Mental Workload Assessment Under Different Levels of Physical Activity , 2019, 2019 9th International IEEE/EMBS Conference on Neural Engineering (NER).

[2]  Daniel P. Ferris,et al.  Removal of movement artifact from high-density EEG recorded during walking and running. , 2010, Journal of neurophysiology.

[3]  Amy Loutfi,et al.  Data Mining for Wearable Sensors in Health Monitoring Systems: A Review of Recent Trends and Challenges , 2013, Sensors.

[4]  Francisco J. Pelayo,et al.  Stress Assessment by Prefrontal Relative Gamma , 2016, Front. Comput. Neurosci..

[5]  M. Frankenhaeuser,et al.  A Psychobiological Framework for Research on Human Stress and Coping , 1986 .

[6]  Manolis Tsiknakis,et al.  Detection of stress/anxiety state from EEG features during video watching , 2015, 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[7]  C. L. Lim,et al.  Human thermoregulation and measurement of body temperature in exercise and clinical settings. , 2008, Annals of the Academy of Medicine, Singapore.

[8]  P. Sloane,et al.  Job Anxiety, Work‐Related Psychological Illness and Workplace Performance , 2016, SSRN Electronic Journal.

[9]  M. Huttunen,et al.  [The nature of stress]. , 1981, Duodecim; laaketieteellinen aikakauskirja.

[10]  D. Carroll,et al.  Heart rate and oxygen consumption during mental arithmetic, a video game, and graded static exercise. , 1987, Psychophysiology.

[11]  P Salmon,et al.  Effects of physical exercise on anxiety, depression, and sensitivity to stress: a unifying theory. , 2001, Clinical psychology review.

[12]  Danilo P. Mandic,et al.  Resolving Ambiguities in the LF/HF Ratio: LF-HF Scatter Plots for the Categorization of Mental and Physical Stress from HRV , 2017, Front. Physiol..

[13]  Malcolm Hobbs,et al.  Measuring manual dexterity and anxiety in divers using a novel task at 35-41 m. , 2012, Aviation, space, and environmental medicine.

[14]  Jean-François Gagnon,et al.  WAUC: A Multi-Modal Database for Mental Workload Assessment Under Physical Activity , 2020, Frontiers in Neuroscience.

[15]  T. Falk,et al.  The effects of automated artifact removal algorithms on electroencephalography-based Alzheimer's disease diagnosis , 2014, Front. Aging Neurosci..

[16]  G. Nappi,et al.  Pupillary and cardiovascular responses to the cold-pressor test. , 1995, Journal of the autonomic nervous system.

[17]  C. Wientjes,et al.  Mental load and work stress as two types of energy mobilization , 1994 .

[18]  Gunar Schirner,et al.  EEGu2: an embedded device for brain/body signal acquisition and processing , 2016, 2016 International Symposium on Rapid System Prototyping (RSP).

[19]  Marco Cecchi,et al.  Towards automated electroencephalography-based Alzheimer's disease diagnosis using portable low-density devices , 2017, Biomed. Signal Process. Control..

[20]  Matjaz Gams,et al.  Continuous stress detection using a wrist device: in laboratory and real life , 2016, UbiComp Adjunct.

[21]  Oliver T. Wolf,et al.  Stress and memory in humans: Twelve years of progress? , 2009, Brain Research.

[22]  Sylvia D. Kreibig,et al.  Cardiovascular, electrodermal, and respiratory response patterns to fear- and sadness-inducing films. , 2007, Psychophysiology.

[23]  Sylvie Charbonnier,et al.  A comparison of wearable and stationary sensors for stress detection , 2016, 2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC).

[24]  Tore Wentzel-Larsen,et al.  Motor impairment in children with anxiety disorders , 2012, Psychiatry Research.

[25]  Jean-François Gagnon,et al.  A Multimodal Approach to Improve the Robustness of Physiological Stress Prediction During Physical Activity , 2019, 2019 IEEE International Conference on Systems, Man and Cybernetics (SMC).

[26]  A. Pichon,et al.  Spectral analysis of heart rate variability during exercise in trained subjects. , 2004, Medicine and science in sports and exercise.

[27]  W. Bardwell,et al.  Effects of stress on heart rate complexity—A comparison between short-term and chronic stress , 2009, Biological Psychology.

[28]  Tiago H. Falk,et al.  On the Analysis of EEG Features for Mental Workload Assessment During Physical Activity , 2018, 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC).

[29]  S. Muthukumaraswamy High-frequency brain activity and muscle artifacts in MEG/EEG: a review and recommendations , 2013, Front. Hum. Neurosci..

[30]  Emre Ertin,et al.  cStress: towards a gold standard for continuous stress assessment in the mobile environment , 2015, UbiComp.

[31]  Oliver T Wolf,et al.  Influence of acute stress on response inhibition in healthy men: An ERP study. , 2017, Psychophysiology.

[32]  Tiago H. Falk,et al.  MuLES: An Open Source EEG Acquisition and Streaming Server for Quick and Simple Prototyping and Recording , 2015, IUI Companion.

[33]  B. Kudielka,et al.  Trier Social Stress Test , 2007 .

[34]  Hugo F Posada-Quintero,et al.  Highly sensitive index of sympathetic activity based on time-frequency spectral analysis of electrodermal activity. , 2016, American journal of physiology. Regulatory, integrative and comparative physiology.

[35]  Hugo F. Posada-Quintero,et al.  Power Spectral Density Analysis of Electrodermal Activity for Sympathetic Function Assessment , 2016, Annals of Biomedical Engineering.

[36]  Olivier Gagnon,et al.  A Systematic Assessment of Operational Metrics for Modeling Operator Functional State , 2016, PhyCS.

[37]  E Donchin,et al.  Brain-computer interface technology: a review of the first international meeting. , 2000, IEEE transactions on rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society.

[38]  Houtan Jebelli,et al.  Measuring Workers’ Emotional State during Construction Tasks Using Wearable EEG , 2018, Journal of Construction Engineering and Management.

[39]  Fabio Babiloni,et al.  Adaptive Automation Triggered by EEG-Based Mental Workload Index: A Passive Brain-Computer Interface Application in Realistic Air Traffic Control Environment , 2016, Front. Hum. Neurosci..

[40]  Tiago H. Falk,et al.  Multi-Scale Heart Beat Entropy Measures for Mental Workload Assessment of Ambulant Users , 2019, Entropy.

[41]  Pascal Carrive,et al.  Changes in cutaneous and body temperature during and after conditioned fear to context in the rat , 2005, The European journal of neuroscience.

[42]  Tiago H. Falk,et al.  Prediction of Stress and Mental Workload during Police Academy Training Using Ultra-Short-Term Heart Rate Variability and Breathing Analysis , 2020, 2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC).

[43]  M Javorka,et al.  Complexity and time asymmetry of heart rate variability are altered in acute mental stress , 2014, Physiological measurement.

[44]  Koushik Maharatna,et al.  Hybrid wavelet and EMD/ICA approach for artifact suppression in pervasive EEG , 2016, Journal of Neuroscience Methods.

[45]  D. Marazziti,et al.  Psychological stress and body temperature changes in humans , 1992, Physiology & Behavior.

[46]  G. Billman Heart Rate Variability – A Historical Perspective , 2011, Front. Physio..

[47]  Angela Didomenico,et al.  Interactive effects of physical and mental workload on subjective workload assessment , 2008 .

[48]  James L. Abelson,et al.  Respiratory variability and sighing: A psychophysiological reset model , 2013, Biological Psychology.

[49]  Elena Smets,et al.  Into the Wild: The Challenges of Physiological Stress Detection in Laboratory and Ambulatory Settings , 2019, IEEE Journal of Biomedical and Health Informatics.

[50]  Martin L. Griss,et al.  Activity-Aware Mental Stress Detection Using Physiological Sensors , 2010, MobiCASE.

[51]  Louis G. Castonguay,et al.  Worry, generalized anxiety disorder, and emotion: Evidence from the EEG gamma band , 2008, Biological Psychology.

[52]  Hugo F Posada-Quintero,et al.  Time-varying analysis of electrodermal activity during exercise , 2018, PloS one.

[53]  B. Saltin,et al.  Exercise as medicine – evidence for prescribing exercise as therapy in 26 different chronic diseases , 2015, Scandinavian journal of medicine & science in sports.

[54]  E. Basar,et al.  Gamma-band responses in the brain: a short review of psychophysiological correlates and functional significance. , 1996, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[55]  P. Sleight,et al.  Physical activity influences heart rate variability and very-low-frequency components in Holter electrocardiograms. , 1996, Cardiovascular research.

[56]  G. Fernández,et al.  Acute Psychological Stress Reduces Working Memory-Related Activity in the Dorsolateral Prefrontal Cortex , 2009, Biological Psychiatry.

[57]  Tiago H. Falk,et al.  Spectro-Temporal Electrocardiogram Analysis for Noise-Robust Heart Rate and Heart Rate Variability Measurement , 2017, IEEE Journal of Translational Engineering in Health and Medicine.

[58]  Syed Muhammad Anwar,et al.  Classification of Perceived Mental Stress Using A Commercially Available EEG Headband , 2019, IEEE Journal of Biomedical and Health Informatics.

[59]  Masayuki Numao,et al.  Towards the Design of Affective Survival Horror Games: An Investigation on Player Affect , 2013, 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction.

[60]  Chad L. Stephens,et al.  Prediction of Cognitive States During Flight Simulation Using Multimodal Psychophysiological Sensing , 2017 .

[61]  K. Brady,et al.  Stress reactivity: biological and subjective responses to the cold pressor and Trier Social stressors , 2006, Human psychopharmacology.

[62]  Christian Grillon,et al.  Working memory performance after acute exposure to the cold pressor stress in healthy volunteers , 2009, Neurobiology of Learning and Memory.

[63]  Tiago H. Falk,et al.  Cross-Subject Statistical Shift Estimation for Generalized Electroencephalography-based Mental Workload Assessment , 2019, 2019 IEEE International Conference on Systems, Man and Cybernetics (SMC).

[64]  Gert Pfurtscheller,et al.  Characterization of four-class motor imagery EEG data for the BCI-competition 2005 , 2005, Journal of neural engineering.

[65]  Mickaël Causse,et al.  The diagnosticity of psychophysiological signatures: Can we disentangle mental workload from acute stress with ECG and fNIRS? , 2019, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[66]  F. Smulders,et al.  The functional role of individual-alpha based frontal asymmetry in stress responding , 2015, Biological Psychology.

[67]  A. Hani,et al.  Mental stress assessment using simultaneous measurement of EEG and fNIRS. , 2016, Biomedical optics express.

[68]  W. Boucsein Electrodermal activity, 2nd ed. , 2012 .

[69]  Maxwell Rainforth,et al.  A self-referential default brain state: patterns of coherence, power, and eLORETA sources during eyes-closed rest and Transcendental Meditation practice , 2010, Cognitive Processing.

[70]  S. Folkman,et al.  Stress, appraisal, and coping , 1974 .

[71]  T. Reinhardt,et al.  Salivary cortisol, heart rate, electrodermal activity and subjective stress responses to the Mannheim Multicomponent Stress Test (MMST) , 2012, Psychiatry Research.

[72]  G. Borghini,et al.  Neuroscience and Biobehavioral Reviews , 2022 .

[73]  Lauren Reinerman-Jones,et al.  Workload Is Multidimensional, Not Unitary: What Now? , 2015, HCI.

[74]  Gerald Matthews,et al.  Dynamic relationships between stress states and working memory , 2010 .

[75]  Martin Maier,et al.  MS-QI: A Modulation Spectrum-Based ECG Quality Index for Telehealth Applications , 2016, IEEE Transactions on Biomedical Engineering.

[76]  Shahjahan Shahid,et al.  Classification of Multichannel EEG Signal by Linear Discriminant Analysis , 2015, ICSEng.

[77]  Neerincx,et al.  EEG alpha asymmetry, heart rate variability and cortisol in response to virtual reality induced stress , 2011 .

[78]  M. Chudecka,et al.  The use of thermal imaging to evaluate body temperature changes of athletes during training and a study on the impact of physiological and morphological factors on skin temperature , 2012 .

[79]  M. Jamal Job stress and job performance controversy revisited: An empirical examination in two countries. , 2007 .

[80]  Ki H. Chon,et al.  Electrodermal Activity Is Sensitive to Cognitive Stress under Water , 2018, Front. Physiol..

[81]  V. A. Makarov,et al.  Recovering EEG brain signals: Artifact suppression with wavelet enhanced independent component analysis , 2006, Journal of Neuroscience Methods.

[82]  Brendan Z. Allison,et al.  Is It Significant? Guidelines for Reporting BCI Performance , 2012 .

[83]  Ricardo Gutierrez-Osuna,et al.  A comparative study of game mechanics and control laws for an adaptive physiological game , 2014, Journal on Multimodal User Interfaces.

[84]  A. Damasio,et al.  Basic emotions are associated with distinct patterns of cardiorespiratory activity. , 2006, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[85]  Joachim A. Behar,et al.  PhysioZoo: A Novel Open Access Platform for Heart Rate Variability Analysis of Mammalian Electrocardiographic Data , 2018, Front. Physiol..

[86]  A. Caspi,et al.  Influence of Life Stress on Depression: Moderation by a Polymorphism in the 5-HTT Gene , 2003, Science.

[87]  Federico Lombardi,et al.  Spectral Analysis of Heart Rate Variability , 1997 .

[88]  J. Lagopoulos Electrodermal activity , 2007, Acta Neuropsychiatrica.

[89]  I. Homma,et al.  Breathing rhythms and emotions , 2008, Experimental physiology.

[90]  Kazuhiro Nakamura,et al.  Central circuitries for body temperature regulation and fever. , 2011, American journal of physiology. Regulatory, integrative and comparative physiology.

[91]  Golda S. Ginsburg,et al.  Results from the Child/Adolescent Anxiety Multimodal Longitudinal Study (CAMELS): Functional Outcomes , 2018, Journal of consulting and clinical psychology.

[92]  Alexandre Campeau-Lecours,et al.  Predicting Video Game Players’ Fun from Physiological and Behavioural Data , 2018, Advances in Intelligent Systems and Computing.

[93]  Daniel P. Ferris,et al.  Independent Component Analysis of Gait-Related Movement Artifact Recorded using EEG Electrodes during Treadmill Walking , 2015, Front. Hum. Neurosci..

[94]  Paula Goolkasian,et al.  Video Games and Stress: How Stress Appraisals and Game Content Affect Cardiovascular and Emotion Outcomes , 2019, Front. Psychol..

[95]  福田 博一 State-Trait Anxiety Inventoryによるペインクリニック外来患者の不安の評価 , 1994 .

[96]  A. K. Blangsted,et al.  The effect of mental stress on heart rate variability and blood pressure during computer work , 2004, European Journal of Applied Physiology.

[97]  N. Schneiderman,et al.  Stress and health: psychological, behavioral, and biological determinants. , 2005, Annual review of clinical psychology.

[98]  Srinivasan Jayaraman,et al.  Electroencephalography Amplitude Modulation Analysis for Automated Affective Tagging of Music Video Clips , 2018, Front. Comput. Neurosci..

[99]  Silvio de Oliveira,et al.  Exergy Analysis of Human Respiration under Physical Activity , 2010 .

[100]  B. Kudielka,et al.  Ten Years of Research with the Trier Social Stress Test--Revisited. , 2007 .

[101]  Sandra G. Hart,et al.  Nasa-Task Load Index (NASA-TLX); 20 Years Later , 2006 .

[102]  Luciano Mecacci,et al.  Anxiety, cognitive self-evaluation and performance: ERP correlates. , 2009, Journal of anxiety disorders.

[103]  Tiago H. Falk,et al.  Performance Comparison of Automated EEG Enhancement Algorithms for Mental Workload Assessment of Ambulant Users , 2019, 2019 9th International IEEE/EMBS Conference on Neural Engineering (NER).

[104]  Desney S. Tan,et al.  Brain-Computer Interfaces and Human-Computer Interaction , 2010, Brain-Computer Interfaces.

[105]  Renato Anghinah,et al.  EEG amplitude modulation analysis for semi-automated diagnosis of Alzheimer’s disease , 2012, EURASIP Journal on Advances in Signal Processing.

[106]  Peter A Hancock,et al.  State of science: mental workload in ergonomics , 2015, Ergonomics.

[107]  Rossana Castaldo,et al.  Acute mental stress assessment via short term HRV analysis in healthy adults: A systematic review with meta-analysis , 2015, Biomed. Signal Process. Control..