EEG-based workload estimation across affective contexts

Workload estimation from electroencephalographic signals (EEG) offers a highly sensitive tool to adapt the human–computer interaction to the user state. To create systems that reliably work in the complexity of the real world, a robustness against contextual changes (e.g., mood), has to be achieved. To study the resilience of state-of-the-art EEG-based workload classification against stress we devise a novel experimental protocol, in which we manipulated the affective context (stressful/non-stressful) while the participant solved a task with two workload levels. We recorded self-ratings, behavior, and physiology from 24 participants to validate the protocol. We test the capability of different, subject-specific workload classifiers using either frequency-domain, time-domain, or both feature varieties to generalize across contexts. We show that the classifiers are able to transfer between affective contexts, though performance suffers independent of the used feature domain. However, cross-context training is a simple and powerful remedy allowing the extraction of features in all studied feature varieties that are more resilient to task-unrelated variations in signal characteristics. Especially for frequency-domain features, across-context training is leading to a performance comparable to within-context training and testing. We discuss the significance of the result for neurophysiology-based workload detection in particular and for the construction of reliable passive brain–computer interfaces in general.

[1]  H. Selye A Syndrome produced by Diverse Nocuous Agents , 1936, Nature.

[2]  Robert J. K. Jacob,et al.  Using fNIRS brain sensing to evaluate information visualization interfaces , 2013, CHI.

[3]  M. Eysenck,et al.  New perspectives in attentional control theory , 2011 .

[4]  Jan B. F. van Erp,et al.  Brain-Based Indices for User System Symbiosis , 2010, Brain-Computer Interfaces.

[5]  Daniel Tranel,et al.  The influence of autonomic arousal and semantic relatedness on memory for emotional words. , 2006, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[6]  Stéphane Bonnet,et al.  Time-on-task effect on workload level discriminability through electroencephalography , 2012 .

[7]  T O Zander,et al.  Context-aware brain–computer interfaces: exploring the information space of user, technical system and environment , 2012, Journal of neural engineering.

[8]  T. Reinhardt,et al.  Salivary cortisol, heart rate, electrodermal activity and subjective stress responses to the Mannheim Multicomponent Stress Test (MMST) , 2012, Psychiatry Research.

[9]  Desney S. Tan,et al.  Feasibility and pragmatics of classifying working memory load with an electroencephalograph , 2008, CHI.

[10]  Michael E. Smith,et al.  Monitoring Working Memory Load during Computer-Based Tasks with EEG Pattern Recognition Methods , 1998, Hum. Factors.

[11]  Christian Mühl,et al.  Flaws in current human training protocols for spontaneous Brain-Computer Interfaces: lessons learned from instructional design , 2013, Front. Hum. Neurosci..

[12]  F.R.H. Zijlstra,et al.  Efficiency in work behaviour: A design approach for modern tools , 1993 .

[13]  Fumio Miyazaki,et al.  Method for Objectively Evaluating Psychological Stress Resulting when Humans Interact with Robots , 2009 .

[14]  Christian Kothe,et al.  Towards passive brain–computer interfaces: applying brain–computer interface technology to human–machine systems in general , 2011, Journal of neural engineering.

[15]  Fabien Lotte,et al.  Brain-Computer Interfaces: Beyond Medical Applications , 2012, Computer.

[16]  Christian Mühl,et al.  Valence, arousal and dominance in the EEG during game play , 2013, Int. J. Auton. Adapt. Commun. Syst..

[17]  Touradj Ebrahimi,et al.  Spatial filters for the classification of event-related potentials , 2006, ESANN.

[18]  Robert Oostenveld,et al.  Estimating workload using EEG spectral power and ERPs in the n-back task , 2012, Journal of neural engineering.

[19]  Matthias Scheutz,et al.  Brainput: enhancing interactive systems with streaming fnirs brain input , 2012, CHI.

[20]  Johannes Hewig,et al.  Associations of the cortisol awakening response (CAR) with cortical activation asymmetry during the course of an exam stress period , 2008, Psychoneuroendocrinology.

[21]  M. Schubert,et al.  The physiological response to Trier Social Stress Test relates to subjective measures of stress during but not before or after the test , 2012, Psychoneuroendocrinology.

[22]  Ad Aertsen,et al.  Review of the BCI Competition IV , 2012, Front. Neurosci..

[23]  R. Krout,et al.  Music listening to facilitate relaxation and promote wellness: Integrated aspects of our neurophysiological responses to music , 2007 .

[24]  Qiang Yang,et al.  A Survey on Transfer Learning , 2010, IEEE Transactions on Knowledge and Data Engineering.

[25]  C. Kirschbaum,et al.  The 'Trier Social Stress Test'--a tool for investigating psychobiological stress responses in a laboratory setting. , 1993, Neuropsychobiology.

[26]  Nicole Y. Weekes,et al.  The effect of a naturalistic stressor on frontal EEG asymmetry, stress, and health , 2007, Biological Psychology.

[27]  Wolfgang Rosenstiel,et al.  Using Cross-Task Classification for Classifying Workload Levels in Complex Learning Tasks , 2013, 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction.

[28]  B. Kudielka,et al.  Trier Social Stress Test , 2007 .

[29]  ChenFang,et al.  Estimating cognitive workload using wavelet entropy-based features during an arithmetic task , 2013 .

[30]  Fuhui Long,et al.  Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy , 2003, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[31]  Clemens Brunner,et al.  Better than random? A closer look on BCI results , 2008 .

[32]  W. Kirchner Age differences in short-term retention of rapidly changing information. , 1958, Journal of experimental psychology.

[33]  Mattie Tops,et al.  State-dependent regulation of cortical activity by cortisol: An EEG study , 2006, Neuroscience Letters.

[34]  Moritz Grosse-Wentrup,et al.  Critical issues in state-of-the-art brain–computer interface signal processing , 2011, Journal of neural engineering.

[35]  Cuntai Guan,et al.  Learning from other subjects helps reducing Brain-Computer Interface calibration time , 2010, 2010 IEEE International Conference on Acoustics, Speech and Signal Processing.

[36]  L. Julian,et al.  Measures of anxiety: State‐Trait Anxiety Inventory (STAI), Beck Anxiety Inventory (BAI), and Hospital Anxiety and Depression Scale‐Anxiety (HADS‐A) , 2011, Arthritis care & research.

[37]  Jan Wacker,et al.  Defensiveness and anxiety predict frontal EEG asymmetry only in specific situational contexts , 2008, Biological Psychology.

[38]  S. Dickerson,et al.  Acute stressors and cortisol responses: a theoretical integration and synthesis of laboratory research. , 2004, Psychological bulletin.

[39]  J. Wolpaw,et al.  EMG contamination of EEG: spectral and topographical characteristics , 2003, Clinical Neurophysiology.

[40]  W. Verwey,et al.  Detecting short periods of elevated workload: A comparison of nine workload assessment techniques , 1996 .

[41]  G. Pfurtscheller,et al.  A fully automated correction method of EOG artifacts in EEG recordings , 2007, Clinical Neurophysiology.

[42]  Christian Mühl,et al.  Design and Validation of a Mental and Social Stress Induction Protocol - Towards Load-invariant Physiology-based Stress Detection , 2014, PhyCS.

[43]  Rajita Sinha,et al.  Hypothalamic-pituitary-adrenal axis and sympatho-adreno-medullary responses during stress-induced and drug cue-induced cocaine craving states , 2003, Psychopharmacology.

[44]  S. Hart,et al.  Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research , 1988 .

[45]  Cuntai Guan,et al.  Filter Bank Common Spatial Pattern Algorithm on BCI Competition IV Datasets 2a and 2b , 2012, Front. Neurosci..

[46]  M Congedo,et al.  A review of classification algorithms for EEG-based brain–computer interfaces , 2007, Journal of neural engineering.

[47]  C. Spielberger,et al.  Manual for the State-Trait Anxiety Inventory , 1970 .

[48]  O. Jensen,et al.  Frontal theta activity in humans increases with memory load in a working memory task , 2002, The European journal of neuroscience.

[49]  Cuntai Guan,et al.  Comparison of designs towards a subject-independent brain-computer interface based on motor imagery , 2009, 2009 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[50]  M. Bradley,et al.  Measuring emotion: the Self-Assessment Manikin and the Semantic Differential. , 1994, Journal of behavior therapy and experimental psychiatry.

[51]  J. Lisman,et al.  Oscillations in the alpha band (9-12 Hz) increase with memory load during retention in a short-term memory task. , 2002, Cerebral cortex.

[52]  Stefan Haufe,et al.  Single-trial analysis and classification of ERP components — A tutorial , 2011, NeuroImage.

[53]  K.-R. Muller,et al.  Optimizing Spatial filters for Robust EEG Single-Trial Analysis , 2008, IEEE Signal Processing Magazine.

[54]  S. Fairclough,et al.  Effects of performance feedback on cardiovascular reactivity and frontal EEG asymmetry. , 2011, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[55]  A. Nijholt,et al.  The Impact of Loss of Control on Movement BCIs , 2011, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[56]  Guillaume Gibert,et al.  OpenViBE: An Open-Source Software Platform to Design, Test, and Use BrainComputer Interfaces in Real and Virtual Environments , 2010, PRESENCE: Teleoperators and Virtual Environments.

[57]  M. Bushnell,et al.  Autonomic responses to heat pain: Heart rate, skin conductance, and their relation to verbal ratings and stimulus intensity , 2011, PAIN®.