Enhanced Performance of a Brain Switch by Simultaneous Use of EEG and NIRS Data for Asynchronous Brain-Computer Interface

Previous studies have shown the superior performance of hybrid electroencephalography (EEG)/ near-infrared spectroscopy (NIRS) brain-computer interfaces (BCIs). However, it has been veiled whether the use of a hybrid EEG/NIRS modality can provide better performance for a brain switch that can detect the onset of the intention to turn on a BCI. In this study, we developed such a hybrid EEG/NIRS brain switch and compared its performance with single modality EEG- and NIRS-based brain switch respectively, in terms of true positive rate (TPR), false positive rate (FPR), onset detection time (ODT), and information transfer rate (ITR). In an offline analysis, the performance of a hybrid EEG/NIRS brain switch was significantly improved over that of EEG- and NIRS-based brain switches in general, and in particular a significantly lower FPR was observed for the hybrid EEG/NIRS brain switch. A pseudo-online analysis was additionally performed to confirm the feasibility of implementing an online BCI system with our hybrid EEG/NIRS brain switch. The overall trend of pseudo-online analysis results generally coincided with that of the offline analysis results. No significant difference in all performance measures was also found between offline and pseudo online analysis schemes when the amount of training data was same, with one exception for the ITRs of an EEG brain switch. These offline and pseudo-online results demonstrate that a hybrid EEG/NIRS brain switch can be used to provide a better onset detection performance than that of a single neuroimaging modality.

[1]  Martin Wolf,et al.  Single-trial classification of motor imagery differing in task complexity: a functional near-infrared spectroscopy study , 2011, Journal of NeuroEngineering and Rehabilitation.

[2]  G. Pfurtscheller,et al.  Self-Paced Operation of an SSVEP-Based Orthosis With and Without an Imagery-Based “Brain Switch:” A Feasibility Study Towards a Hybrid BCI , 2010, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[3]  Clemens Brunner,et al.  Analysis of sensorimotor rhythms for the implementation of a brain switch for healthy subjects , 2010, Biomed. Signal Process. Control..

[4]  Vera Kaiser,et al.  Fast set-up asynchronous brain-switch based on detection of foot motor imagery in 1-channel EEG , 2010, Medical & Biological Engineering & Computing.

[5]  Stefan Haufe,et al.  Optimizing event-related potential based brain-computer interfaces: a systematic evaluation of dynamic stopping methods. , 2013, Journal of neural engineering.

[6]  Keum-Shik Hong,et al.  Passive BCI based on drowsiness detection: an fNIRS study. , 2015, Biomedical optics express.

[7]  C. Im,et al.  Assessment of user voluntary engagement during neurorehabilitation using functional near-infrared spectroscopy: a preliminary study , 2018, Journal of NeuroEngineering and Rehabilitation.

[8]  E. Sellers,et al.  How many people are able to control a P300-based brain–computer interface (BCI)? , 2009, Neuroscience Letters.

[9]  Klaus-Robert Müller,et al.  Learning From More Than One Data Source: Data Fusion Techniques for Sensorimotor Rhythm-Based Brain–Computer Interfaces , 2015, Proceedings of the IEEE.

[10]  Chang-Hwan Im,et al.  Evaluation of various mental task combinations for near-infrared spectroscopy-based brain-computer interfaces , 2014, Journal of biomedical optics.

[11]  Klaus-Robert Müller,et al.  Improvement of Information Transfer Rates Using a Hybrid EEG-NIRS Brain-Computer Interface with a Short Trial Length: Offline and Pseudo-Online Analyses , 2018, Sensors.

[12]  John Q. Gan,et al.  Unsupervised movement onset detection from EEG recorded during self-paced real hand movement , 2010, Medical & Biological Engineering & Computing.

[13]  Gert Pfurtscheller,et al.  Motor imagery and direct brain-computer communication , 2001, Proc. IEEE.

[14]  Mads Jochumsen,et al.  Continuous 2D control via state-machine triggered by endogenous sensory discrimination and a fast brain switch , 2019, Journal of neural engineering.

[15]  Tom Holroyd,et al.  A High Performance MEG Based BCI Using Single Trial Detection of Human Movement Intention , 2013 .

[16]  Cuntai Guan,et al.  Temporal classification of multichannel near-infrared spectroscopy signals of motor imagery for developing a brain–computer interface , 2007, NeuroImage.

[17]  Wolfgang Rosenstiel,et al.  An MEG-based brain–computer interface (BCI) , 2007, NeuroImage.

[18]  Shirley M Coyle,et al.  Brain–computer interface using a simplified functional near-infrared spectroscopy system , 2007, Journal of neural engineering.

[19]  Christa Neuper,et al.  Single-trial classification of antagonistic oxyhemoglobin responses during mental arithmetic , 2011, Medical & Biological Engineering & Computing.

[20]  Piotr Stawicki,et al.  Age-related differences in SSVEP-based BCI performance , 2017, Neurocomputing.

[21]  Lin Zhang,et al.  An FDES-Based Shared Control Method for Asynchronous Brain-Actuated Robot , 2016, IEEE Transactions on Cybernetics.

[22]  Christa Neuper,et al.  Focal frontal (de)oxyhemoglobin responses during simple arithmetic. , 2010, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[23]  Shiuan Huang,et al.  Novel Motor Imagery-Based Brain Switch for Patients With Amyotrophic Lateral Sclerosis: A Case Study Using Two-Channel Electroencephalography , 2019, IEEE Consumer Electronics Magazine.

[24]  Brendan Z. Allison,et al.  The Hybrid BCI , 2010, Frontiers in Neuroscience.

[25]  Yuanqing Li,et al.  Control of a Wheelchair in an Indoor Environment Based on a Brain–Computer Interface and Automated Navigation , 2016, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[26]  T. Chau,et al.  Towards a system-paced near-infrared spectroscopy brain–computer interface: differentiating prefrontal activity due to mental arithmetic and mental singing from the no-control state , 2011, Journal of neural engineering.

[27]  Yuliya Zaytseva,et al.  EEG correlates of a mental arithmetic task in patients with first episode schizophrenia and schizoaffective disorder , 2015, Clinical Neurophysiology.

[28]  H. Levene.  Robust tests for equality of variances , 1961 .

[29]  Wolfgang Grodd,et al.  Principles of a brain-computer interface (BCI) based on real-time functional magnetic resonance imaging (fMRI) , 2004, IEEE Transactions on Biomedical Engineering.

[30]  Klaus-Robert Müller,et al.  Neurophysiological predictor of SMR-based BCI performance , 2010, NeuroImage.

[31]  Klaus-Robert Müller,et al.  Classifying Single Trial EEG: Towards Brain Computer Interfacing , 2001, NIPS.

[32]  Christa Neuper,et al.  Temporal Coding of Brain Patterns for Direct Limb Control in Humans , 2010, Front. Neurosci..

[33]  Rui Zhang,et al.  Control of a nursing bed based on a hybrid brain-computer interface , 2016, 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[34]  T. Chau,et al.  Automatic detection of a prefrontal cortical response to emotionally rated music using multi-channel near-infrared spectroscopy , 2012, Journal of neural engineering.

[35]  Young-Joo Kim,et al.  EEG Based Comparative Measurement of Visual Fatigue Caused by 2D and 3D Displays , 2011, HCI.

[36]  Chang-Hwan Im,et al.  An emergency call system for patients in locked-in state using an SSVEP-based brain switch. , 2017, Psychophysiology.

[37]  Feng Li,et al.  Discrimination Between Control and Idle States in Asynchronous SSVEP-Based Brain Switches: A Pseudo-Key-Based Approach , 2013, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[38]  Yuanqing Li,et al.  A P300-Based Threshold-Free Brain Switch and Its Application in Wheelchair Control , 2017, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[39]  Klaus-Robert Müller,et al.  Brain-Switches for Asynchronous Brain–Computer Interfaces: A Systematic Review , 2020 .

[40]  Sarah D Power,et al.  Classification of prefrontal activity due to mental arithmetic and music imagery using hidden Markov models and frequency domain near-infrared spectroscopy , 2010, Journal of neural engineering.

[41]  Chang-Hwan Im,et al.  Development of an SSVEP-based BCI spelling system adopting a QWERTY-style LED keyboard , 2012, Journal of Neuroscience Methods.

[42]  S. Shapiro,et al.  An Approximate Analysis of Variance Test for Normality , 1972 .

[43]  T. Chau,et al.  Single-trial classification of NIRS signals during emotional induction tasks: towards a corporeal machine interface , 2009, Journal of NeuroEngineering and Rehabilitation.

[44]  Niels Birbaumer,et al.  fMRI Brain-Computer Interface: A Tool for Neuroscientific Research and Treatment , 2007, Comput. Intell. Neurosci..

[45]  Andrej M. Savić,et al.  Feasibility of a Hybrid Brain-Computer Interface for Advanced Functional Electrical Therapy , 2014, TheScientificWorldJournal.

[46]  Febo Cincotti,et al.  Towards Noninvasive Hybrid Brain–Computer Interfaces: Framework, Practice, Clinical Application, and Beyond , 2015, Proceedings of the IEEE.

[47]  E Donchin,et al.  Brain-computer interface technology: a review of the first international meeting. , 2000, IEEE transactions on rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society.

[48]  K.-R. Muller,et al.  Optimizing Spatial filters for Robust EEG Single-Trial Analysis , 2008, IEEE Signal Processing Magazine.

[49]  Seungjin Choi,et al.  A hybrid NIRS-EEG system for self-paced brain computer interface with online motor imagery , 2015, Journal of Neuroscience Methods.

[50]  Klaus-Robert Müller,et al.  Descriptor : Simultaneous acquisition of EEG and NIRS during cognitive tasks for an open access dataset , 2018 .

[51]  S. Grimm,et al.  Frontal midline theta oscillations during mental arithmetic: effects of stress , 2015, Front. Behav. Neurosci..

[52]  Jin Uk Kwon,et al.  A Ternary Hybrid EEG-NIRS Brain-Computer Interface for the Classification of Brain Activation Patterns during Mental Arithmetic, Motor Imagery, and Idle State , 2018, Front. Neuroinform..

[53]  K.-R. Muller,et al.  Boosting bit rates and error detection for the classification of fast-paced motor commands based on single-trial EEG analysis , 2003, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[54]  Klaus-Robert Müller,et al.  Open Access Dataset for EEG+NIRS Single-Trial Classification , 2017, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[55]  Chuang Lin,et al.  Endogenous sensory discrimination and selection by a fast brain switch for a high transfer rate brain-computer interface , 2016, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[56]  Wolfram Burgard,et al.  Deep learning with convolutional neural networks for EEG decoding and visualization , 2017, Human brain mapping.

[57]  J. Doyle,et al.  Electroencephalogram correlates of higher cortical functions. , 1979, Science.

[58]  Zoran Nenadic,et al.  Electroencephalography-based endogenous brain–computer interface for online communication with a completely locked-in patient , 2019, Journal of NeuroEngineering and Rehabilitation.

[59]  Han-Jeong Hwang,et al.  Near-infrared spectroscopy (NIRS)-based eyes-closed brain-computer interface (BCI) using prefrontal cortex activation due to mental arithmetic , 2016, Scientific Reports.

[60]  G. Pfurtscheller,et al.  An SSVEP BCI to Control a Hand Orthosis for Persons With Tetraplegia , 2011, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[61]  G. Pfurtscheller,et al.  Could the beta rebound in the EEG be suitable to realize a “brain switch”? , 2009, Clinical Neurophysiology.

[62]  M. Tangermann,et al.  Automatic Classification of Artifactual ICA-Components for Artifact Removal in EEG Signals , 2011, Behavioral and Brain Functions.

[63]  J. Mourino,et al.  Asynchronous BCI and local neural classifiers: an overview of the adaptive brain interface project , 2003, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[64]  Soo-Young Lee,et al.  Brain–computer interface using fMRI: spatial navigation by thoughts , 2004, Neuroreport.

[65]  Klaus-Robert Müller,et al.  Enhanced Performance by a Hybrid Nirs–eeg Brain Computer Interface , 2022 .

[66]  Chang-Hwan Im,et al.  A vision-free brain-computer interface (BCI) paradigm based on auditory selective attention , 2011, 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[67]  Mikhail A. Lebedev,et al.  Neurofeedback Therapy for Enhancing Visual Attention: State-of-the-Art and Challenges , 2016, Front. Neurosci..

[68]  Stephanie Brandl,et al.  Robust artifactual independent component classification for BCI practitioners , 2014, Journal of neural engineering.

[69]  Andrei V. Medvedev,et al.  Does the resting state connectivity have hemispheric asymmetry? A near-infrared spectroscopy study , 2014, NeuroImage.

[70]  Jie Li,et al.  Asynchronous Brain–Computer Interface Based on Steady-State Visual-Evoked Potential , 2013, Cognitive Computation.

[71]  K. Jellinger.  Toward Brain-Computer Interfacing , 2009 .

[72]  J. Friedman.  Regularized Discriminant Analysis , 1989 .

[73]  Andrzej Cichocki,et al.  Bimodal BCI Using Simultaneously NIRS and EEG , 2014, IEEE Transactions on Biomedical Engineering.

[74]  Gary E. Birch,et al.  Brain-computer interface design for asynchronous control applications: improvements to the LF-ASD asynchronous brain switch , 2004, IEEE Transactions on Biomedical Engineering.

[75]  Hasan Onur Keles,et al.  Hybrid EEG-fNIRS Asynchronous Brain-Computer Interface for Multiple Motor Tasks , 2016, PloS one.

[76]  Klaus-Robert Müller,et al.  Interpretable deep neural networks for single-trial EEG classification , 2016, Journal of Neuroscience Methods.

[77]  Cuntai Guan,et al.  Asynchronous P300-Based Brain--Computer Interfaces: A Computational Approach With Statistical Models , 2008, IEEE Transactions on Biomedical Engineering.

[78]  T. Chau,et al.  Effects of user mental state on EEG-BCI performance , 2015, Front. Hum. Neurosci..

[79]  Filippo Zappasodi,et al.  Deep learning for hybrid EEG-fNIRS brain–computer interface: application to motor imagery classification , 2018, Journal of neural engineering.

[80]  Stefan Haufe,et al.  The Berlin Brain-Computer Interface: Progress Beyond Communication and Control , 2016, Front. Neurosci..

[81]  Klaus-Robert Müller,et al.  Evaluation of a Compact Hybrid Brain-Computer Interface System , 2017, BioMed research international.

[82]  Dennis J. McFarland,et al.  Brain–computer interfaces for communication and control , 2002, Clinical Neurophysiology.

[83]  Gert Pfurtscheller,et al.  Development, set-up and first results for a one-channel near-infrared spectroscopy system / Entwicklung, Aufbau und vorläufige Ergebnisse eines Einkanal- Nahinfrarot-Spektroskopie-Systems , 2008, Biomedizinische Technik. Biomedical engineering.

[84]  Gernot R. Müller-Putz,et al.  Self-Paced (Asynchronous) BCI Control of a Wheelchair in Virtual Environments: A Case Study with a Tetraplegic , 2007, Comput. Intell. Neurosci..

[85]  C. Jutten,et al.  A Brain-Switch using Riemannian Geometry , 2011 .

[86]  Klaus-Robert Müller,et al.  The non-invasive Berlin Brain–Computer Interface: Fast acquisition of effective performance in untrained subjects , 2007, NeuroImage.

[87]  Klaus-Robert Müller,et al.  Covariate Shift Adaptation by Importance Weighted Cross Validation , 2007, J. Mach. Learn. Res..

[88]  M. Sams,et al.  EEG and MEG brain-computer interface for tetraplegic patients , 2006, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[89]  Chang-Hwan Im,et al.  Development of a Brain–Computer Interface Toggle Switch with Low False-Positive Rate Using Respiration-Modulated Photoplethysmography , 2020, Sensors.