Estimating Cognitive Workload in an Interactive Virtual Reality Environment Using EEG

With the recent surge of affordable, high-performance virtual reality (VR) headsets, there is unlimited potential for applications ranging from education, to training, to entertainment, to fitness and beyond. As these interfaces continue to evolve, passive user-state monitoring can play a key role in expanding the immersive VR experience, and tracking activity for user well-being. By recording physiological signals such as the electroencephalogram (EEG) during use of a VR device, the user's interactions in the virtual environment could be adapted in real-time based on the user's cognitive state. Current VR headsets provide a logical, convenient, and unobtrusive framework for mounting EEG sensors. The present study evaluates the feasibility of passively monitoring cognitive workload via EEG while performing a classical n-back task in an interactive VR environment. Data were collected from 15 participants and the spatio-spectral EEG features were analyzed with respect to task performance. The results indicate that scalp measurements of electrical activity can effectively discriminate three workload levels, even after suppression of a co-varying high-frequency activity.

[1]  Tanja Schultz,et al.  Mental workload during n-back task—quantified in the prefrontal cortex using fNIRS , 2014, Front. Hum. Neurosci..

[2]  Jacek Gwizdka,et al.  Using Wireless EEG Signals to Assess Memory Workload in the $n$-Back Task , 2016, IEEE Transactions on Human-Machine Systems.

[3]  J. E. Korteling,et al.  Using neurophysiological signals that reflect cognitive or affective state: six recommendations to avoid common pitfalls , 2015, Front. Neurosci..

[4]  Marko Horvat,et al.  Assessing emotional responses induced in virtual reality using a consumer EEG headset: A preliminary report , 2018, 2018 41st International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO).

[5]  M.C. Cavusoglu,et al.  A Detection Scheme for Frontalis and Temporalis Muscle EMG Contamination of EEG Data , 2006, 2006 International Conference of the IEEE Engineering in Medicine and Biology Society.

[6]  K. Pope,et al.  Cognitive tasks augment gamma EEG power , 2004, Clinical Neurophysiology.

[7]  Robert Oostenveld,et al.  Estimating workload using EEG spectral power and ERPs in the n-back task , 2012, Journal of neural engineering.

[8]  Hans-Jochen Heinze,et al.  Systematic comparison between a wireless EEG system with dry electrodes and a wired EEG system with wet electrodes , 2019, NeuroImage.

[9]  Mikael Forsman,et al.  Consistency in physiological stress responses and electromyographic activity during induced stress exposure in women and men , 2004, Integrative physiological and behavioral science : the official journal of the Pavlovian Society.

[10]  Síria Monyelle Silva de Oliveira,et al.  Electroencephalographic changes using virtual reality program: technical note , 2018, Neurological research.

[11]  Fotis Liarokapis,et al.  Examining the effect of body ownership in immersive virtual and augmented reality environments , 2016, The Visual Computer.

[12]  Suzanne T. Witt,et al.  Self-paced working memory: Validation of verbal variations of the n-back paradigm , 2007, Brain Research.

[13]  Sebastian Grissmann,et al.  Electroencephalography Based Analysis of Working Memory Load and Affective Valence in an N-back Task with Emotional Stimuli , 2017, Front. Hum. Neurosci..

[14]  Sarah D Power,et al.  Classification of prefrontal activity due to mental arithmetic and music imagery using hidden Markov models and frequency domain near-infrared spectroscopy , 2010, Journal of neural engineering.

[15]  Thorsten O. Zander,et al.  Using neurophysiological signals that reflect cognitive or affective state , 2015 .

[16]  Sebastian Grissmann,et al.  Context Sensitivity of EEG-Based Workload Classification Under Different Affective Valence , 2020, IEEE Transactions on Affective Computing.

[17]  Kilseop Ryu,et al.  Evaluation of mental workload with a combined measure based on physiological indices during a dual task of tracking and mental arithmetic , 2005 .

[18]  J. H. Clark The Ishihara Test for Color Blindness. , 1924 .

[19]  Daniel P. Ferris,et al.  Independent Component Analysis of Gait-Related Movement Artifact Recorded using EEG Electrodes during Treadmill Walking , 2015, Front. Hum. Neurosci..

[20]  Kenneth J. Pope,et al.  Thinking activates EMG in scalp electrical recordings , 2008, Clinical Neurophysiology.

[21]  Carryl L. Baldwin,et al.  Adaptive training using an artificial neural network and EEG metrics for within- and cross-task workload classification , 2012, NeuroImage.

[22]  Jürgen Bergeler,et al.  Light-cured polymer electrodes for non-invasive EEG recordings , 2018, Scientific Reports.

[23]  Daniel P Ferris,et al.  Isolating gait-related movement artifacts in electroencephalography during human walking , 2015, Journal of neural engineering.

[24]  D Roman-Liu,et al.  The influence of mental load on muscle tension , 2013, Ergonomics.

[25]  A. Doud,et al.  Continuous Three-Dimensional Control of a Virtual Helicopter Using a Motor Imagery Based Brain-Computer Interface , 2011, PloS one.

[26]  T. Schubert,et al.  Comparison of the Working Memory Load in N-Back and Working Memory Span Tasks by Means of EEG Frequency Band Power and P300 Amplitude , 2017, Front. Hum. Neurosci..

[27]  Gerald Matthews,et al.  Individual differences in energetic arousal and sustained attention: a dual-task study , 2001 .

[28]  Gabriel Curio,et al.  Unsupervised classification of operator workload from brain signals. , 2016, Journal of neural engineering.

[29]  Kathryn M. McMillan,et al.  N‐back working memory paradigm: A meta‐analysis of normative functional neuroimaging studies , 2005, Human brain mapping.

[30]  Wolfgang Rosenstiel,et al.  Cognitive state monitoring and the design of adaptive instruction in digital environments: lessons learned from cognitive workload assessment using a passive brain-computer interface approach , 2014, Front. Neurosci..

[31]  Maarten A. Hogervorst,et al.  Combining and comparing EEG, peripheral physiology and eye-related measures for the assessment of mental workload , 2014, Front. Neurosci..

[32]  Tanja Schultz,et al.  Classification of mental tasks in the prefrontal cortex using fNIRS , 2013, 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[33]  S. Muthukumaraswamy High-frequency brain activity and muscle artifacts in MEG/EEG: a review and recommendations , 2013, Front. Hum. Neurosci..

[34]  N. Birbaumer,et al.  BCI2000: a general-purpose brain-computer interface (BCI) system , 2004, IEEE Transactions on Biomedical Engineering.

[35]  Dirk Heylen,et al.  Brain-Computer Interfacing and Games , 2010, Brain-Computer Interfaces.

[36]  Anatole Lécuyer,et al.  Exploring Large Virtual Environments by Thoughts Using a BrainComputer Interface Based on Motor Imagery and High-Level Commands , 2010, PRESENCE: Teleoperators and Virtual Environments.

[37]  Christian Mühl,et al.  EEG-based workload estimation across affective contexts , 2014, Front. Neurosci..

[38]  Ricardo Ron-Angevin,et al.  Brain–computer interface: Changes in performance using virtual reality techniques , 2009, Neuroscience Letters.

[39]  H. Lüders,et al.  American Electroencephalographic Society Guidelines for Standard Electrode Position Nomenclature , 1991, Journal of clinical neurophysiology : official publication of the American Electroencephalographic Society.

[40]  Pekcan Ungan,et al.  EEG-like signals can be synthesized from surface representations of single motor units of facial muscles , 2018, Experimental Brain Research.

[41]  Horst Bischof,et al.  Toward Self-Paced Brain–Computer Communication: Navigation Through Virtual Worlds , 2008, IEEE Transactions on Biomedical Engineering.

[42]  G. Riva,et al.  The Past, Present, and Future of Virtual and Augmented Reality Research: A Network and Cluster Analysis of the Literature , 2018, Front. Psychol..

[43]  Stephen H. Fairclough,et al.  Evaluation of an Adaptive Game that Uses EEG Measures Validated during the Design Process as Inputs to a Biocybernetic Loop , 2016, Front. Hum. Neurosci..

[44]  Kemal S. Türker,et al.  Interference of tonic muscle activity on the EEG: a single motor unit study , 2014, Front. Hum. Neurosci..

[45]  Benjamin Wittevrongel,et al.  N‐back training and transfer effects revealed by behavioral responses and EEG , 2018, Brain and behavior.

[46]  Anker Helms Jørgensen,et al.  Effect of mental and physical demands on muscular activity during the use of a computer mouse and a keyboard. , 2002, Scandinavian journal of work, environment & health.

[47]  Christian Kothe,et al.  Towards passive brain–computer interfaces: applying brain–computer interface technology to human–machine systems in general , 2011, Journal of neural engineering.

[48]  Laehyun Kim,et al.  Cross-association analysis of EEG and EMG signals according to movement intention state , 2017, Cognitive Systems Research.

[49]  Hasan Ayaz,et al.  Optical brain monitoring for operator training and mental workload assessment , 2012, NeuroImage.

[50]  Isabela Granic,et al.  Designing and Utilizing Biofeedback Games for Emotion Regulation: The Case of Nevermind , 2016, CHI Extended Abstracts.

[51]  Anil Kumar,et al.  Dry Electrode-Based Fully Isolated EEG/fNIRS Hybrid Brain-Monitoring System , 2019, IEEE Transactions on Biomedical Engineering.

[52]  D. Ruchkin,et al.  Event-related potentials during arithmetic and mental rotation. , 1991, Electroencephalography and clinical neurophysiology.

[53]  Daniel P. Ferris,et al.  Removal of movement artifact from high-density EEG recorded during walking and running. , 2010, Journal of neurophysiology.

[54]  Reinhold Scherer,et al.  FORCe: Fully Online and Automated Artifact Removal for Brain-Computer Interfacing , 2015, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[55]  Catherine Tallon-Baudry,et al.  Induced γ-Band Activity during the Delay of a Visual Short-Term Memory Task in Humans , 1998, The Journal of Neuroscience.

[56]  Gernot R. Müller-Putz,et al.  Self-Paced (Asynchronous) BCI Control of a Wheelchair in Virtual Environments: A Case Study with a Tetraplegic , 2007, Comput. Intell. Neurosci..

[57]  Robert J. K. Jacob,et al.  Designing a passive brain computer interface using real time classification of functional near-infrared spectroscopy , 2013, Int. J. Auton. Adapt. Commun. Syst..

[58]  Meir Plotnik,et al.  Performance-based approach for movement artifact removal from electroencephalographic data recorded during locomotion , 2018, PloS one.

[59]  Michelle N. Lumicao,et al.  EEG correlates of task engagement and mental workload in vigilance, learning, and memory tasks. , 2007, Aviation, space, and environmental medicine.

[60]  Michitaka Hirose,et al.  Brain-Computer Interfaces, Virtual Reality, and Videogames , 2008, Computer.

[61]  Thorsten O. Zander,et al.  Evaluation of a Dry EEG System for Application of Passive Brain-Computer Interfaces in Autonomous Driving , 2017, Front. Hum. Neurosci..

[62]  J. F. Mackworth Vigilance, arousal, and habituation. , 1968, Psychological review.

[63]  W. Kirchner Age differences in short-term retention of rapidly changing information. , 1958, Journal of experimental psychology.

[64]  F. Lotte,et al.  Self-Paced Brain-Computer Interaction with Virtual Worlds: A Quantitative and Qualitative Study "Out of the Lab" , 2008 .

[65]  T. Chau,et al.  Intersession Consistency of Single-Trial Classification of the Prefrontal Response to Mental Arithmetic and the No-Control State by NIRS , 2012, PloS one.

[66]  W. Klimesch,et al.  Control mechanisms in working memory: A possible function of EEG theta oscillations , 2010, Neuroscience & Biobehavioral Reviews.

[67]  Meike Jipp,et al.  Assessing the Driver’s Current Level of Working Memory Load with High Density Functional Near-infrared Spectroscopy: A Realistic Driving Simulator Study , 2017, Front. Hum. Neurosci..

[68]  Dirk Heylen,et al.  Bacteria Hunt , 2010, Journal on Multimodal User Interfaces.

[69]  J. Golding Predicting individual differences in motion sickness susceptibility by questionnaire , 2006 .

[70]  Thorsten O. Zander,et al.  Combining Eye Gaze Input With a Brain–Computer Interface for Touchless Human–Computer Interaction , 2010, Int. J. Hum. Comput. Interact..

[71]  P. Uhlhaas,et al.  Working memory and neural oscillations: alpha–gamma versus theta–gamma codes for distinct WM information? , 2014, Trends in Cognitive Sciences.

[72]  Rosa H. M. Chan,et al.  An evaluation of mental workload with frontal EEG , 2017, PloS one.

[73]  Bin He,et al.  EEG Control of a Virtual Helicopter in 3-Dimensional Space Using Intelligent Control Strategies , 2010, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[74]  Manuel Schabus,et al.  Fronto-parietal EEG coherence in theta and upper alpha reflect central executive functions of working memory. , 2005, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[75]  Kenneth J. Pope,et al.  Evaluation of a minimum-norm based beamforming technique, sLORETA, for reducing tonic muscle contamination of EEG at sensor level , 2017, Journal of Neuroscience Methods.

[76]  Joel S. Warm,et al.  Vigilance Requires Hard Mental Work and Is Stressful , 2008, Hum. Factors.

[77]  Mirka Pesonen,et al.  Brain oscillatory 4–30 Hz responses during a visual n-back memory task with varying memory load , 2007, Brain Research.

[78]  J. Wolpaw,et al.  EMG contamination of EEG: spectral and topographical characteristics , 2003, Clinical Neurophysiology.

[79]  Meike Jipp,et al.  Recognizing Frustration of Drivers From Face Video Recordings and Brain Activation Measurements With Functional Near-Infrared Spectroscopy , 2018, Front. Hum. Neurosci..

[80]  Francisco Velasco-Álvarez,et al.  Free Virtual Navigation Using Motor Imagery Through an Asynchronous BrainComputer Interface , 2010, PRESENCE: Teleoperators and Virtual Environments.

[81]  Fotis Liarokapis,et al.  EEG-based BCI and video games: a progress report , 2018, Virtual Reality.

[82]  G. Mirka,et al.  Cervicobrachial muscle response to cognitive load in a dual-task scenario , 2004, Ergonomics.

[83]  Heidi Sveistrup,et al.  Functional Balance and Dual-Task Reaction Times in Older Adults Are Improved by Virtual Reality and Biofeedback Training , 2007, Cyberpsychology Behav. Soc. Netw..

[84]  Cleotilde Gonzalez,et al.  Task Workload and Cognitive Abilities in Dynamic Decision Making , 2005, Hum. Factors.