Improvement of Information Transfer Rates Using a Hybrid EEG-NIRS Brain-Computer Interface with a Short Trial Length: Offline and Pseudo-Online Analyses

Electroencephalography (EEG) and near-infrared spectroscopy (NIRS) are non-invasive neuroimaging methods that record the electrical and metabolic activity of the brain, respectively. Hybrid EEG-NIRS brain-computer interfaces (hBCIs) that use complementary EEG and NIRS information to enhance BCI performance have recently emerged to overcome the limitations of existing unimodal BCIs, such as vulnerability to motion artifacts for EEG-BCI or low temporal resolution for NIRS-BCI. However, with respect to NIRS-BCI, in order to fully induce a task-related brain activation, a relatively long trial length (≥10 s) is selected owing to the inherent hemodynamic delay that lowers the information transfer rate (ITR; bits/min). To alleviate the ITR degradation, we propose a more practical hBCI operated by intuitive mental tasks, such as mental arithmetic (MA) and word chain (WC) tasks, performed within a short trial length (5 s). In addition, the suitability of the WC as a BCI task was assessed, which has so far rarely been used in the BCI field. In this experiment, EEG and NIRS data were simultaneously recorded while participants performed MA and WC tasks without preliminary training and remained relaxed (baseline; BL). Each task was performed for 5 s, which was a shorter time than previous hBCI studies. Subsequently, a classification was performed to discriminate MA-related or WC-related brain activations from BL-related activations. By using hBCI in the offline/pseudo-online analyses, average classification accuracies of 90.0 ± 7.1/85.5 ± 8.1% and 85.8 ± 8.6/79.5 ± 13.4% for MA vs. BL and WC vs. BL, respectively, were achieved. These were significantly higher than those of the unimodal EEG- or NIRS-BCI in most cases. Given the short trial length and improved classification accuracy, the average ITRs were improved by more than 96.6% for MA vs. BL and 87.1% for WC vs. BL, respectively, compared to those reported in previous studies. The suitability of implementing a more practical hBCI based on intuitive mental tasks without preliminary training and with a shorter trial length was validated when compared to previous studies.

[1]  Ricardo Chavarriaga,et al.  A hybrid brain–computer interface based on the fusion of electroencephalographic and electromyographic activities , 2011, Journal of neural engineering.

[2]  Christa Neuper,et al.  Single-trial classification of antagonistic oxyhemoglobin responses during mental arithmetic , 2011, Medical & Biological Engineering & Computing.

[3]  K.-R. Muller,et al.  Optimizing Spatial filters for Robust EEG Single-Trial Analysis , 2008, IEEE Signal Processing Magazine.

[4]  Klaus-Robert Müller,et al.  Neurophysiological predictor of SMR-based BCI performance , 2010, NeuroImage.

[5]  Stefan Haufe,et al.  Multivariate Machine Learning Methods for Fusing Multimodal Functional Neuroimaging Data , 2015, Proceedings of the IEEE.

[6]  Wolfgang Grodd,et al.  Principles of a brain-computer interface (BCI) based on real-time functional magnetic resonance imaging (fMRI) , 2004, IEEE Transactions on Biomedical Engineering.

[7]  Yunfa Fu,et al.  A hybrid BCI based on EEG and fNIRS signals improves the performance of decoding motor imagery of both force and speed of hand clenching , 2015, Journal of neural engineering.

[8]  Olivier Ledoit,et al.  A well-conditioned estimator for large-dimensional covariance matrices , 2004 .

[9]  Brendan Z. Allison,et al.  The Hybrid BCI , 2010, Frontiers in Neuroscience.

[10]  Karl J. Friston,et al.  Nonlinear Responses in fMRI: The Balloon Model, Volterra Kernels, and Other Hemodynamics , 2000, NeuroImage.

[11]  Arnaud Delorme,et al.  EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis , 2004, Journal of Neuroscience Methods.

[12]  Siamac Fazli,et al.  Brain Computer Interfacing: A Multi-Modal Perspective , 2013, J. Comput. Sci. Eng..

[13]  E. Curran,et al.  Learning to control brain activity: A review of the production and control of EEG components for driving brain–computer interface (BCI) systems , 2003, Brain and Cognition.

[14]  T. Chau,et al.  Towards a system-paced near-infrared spectroscopy brain–computer interface: differentiating prefrontal activity due to mental arithmetic and mental singing from the no-control state , 2011, Journal of neural engineering.

[15]  Mark S. Cohen,et al.  Simultaneous EEG and fMRI of the alpha rhythm , 2002, Neuroreport.

[16]  Reza Fazel-Rezai,et al.  A Review of Hybrid Brain-Computer Interface Systems , 2013, Adv. Hum. Comput. Interact..

[17]  C Neuper,et al.  A comparison of three brain–computer interfaces based on event-related desynchronization, steady state visual evoked potentials, or a hybrid approach using both signals , 2011, Journal of neural engineering.

[18]  Keum-Shik Hong,et al.  Detection and classification of three-class initial dips from prefrontal cortex. , 2017, Biomedical optics express.

[19]  Karim Jerbi,et al.  Exceeding chance level by chance: The caveat of theoretical chance levels in brain signal classification and statistical assessment of decoding accuracy , 2015, Journal of Neuroscience Methods.

[20]  G Pfurtscheller,et al.  Toward a hybrid brain–computer interface based on imagined movement and visual attention , 2010, Journal of neural engineering.

[21]  Chang-Hwan Im,et al.  Performance enhancement of a brain-computer interface using high-density multi-distance NIRS , 2017, Scientific Reports.

[22]  T. Chau,et al.  Single-trial classification of NIRS signals during emotional induction tasks: towards a corporeal machine interface , 2009, Journal of NeuroEngineering and Rehabilitation.

[23]  Keum Shik Hong,et al.  Hybrid EEG–fNIRS-Based Eight-Command Decoding for BCI: Application to Quadcopter Control , 2017, Front. Neurorobot..

[24]  A. Eke,et al.  The modified Beer–Lambert law revisited , 2006, Physics in medicine and biology.

[25]  Seungjin Choi,et al.  A hybrid NIRS-EEG system for self-paced brain computer interface with online motor imagery , 2015, Journal of Neuroscience Methods.

[26]  M. Herrmann,et al.  Frontal activation during a verbal-fluency task as measured by near-infrared spectroscopy , 2003, Brain Research Bulletin.

[27]  Xu Cui,et al.  Speeded Near Infrared Spectroscopy (NIRS) Response Detection , 2010, PloS one.

[28]  Sarah D Power,et al.  Automatic single-trial discrimination of mental arithmetic, mental singing and the no-control state from prefrontal activity: toward a three-state NIRS-BCI , 2012, BMC Research Notes.

[29]  Richard A. Mason,et al.  Prefrontal activation during verbal fluency tests in schizophrenia—a near-infrared spectroscopy (NIRS) study , 2005, Schizophrenia Research.

[30]  Keum-Shik Hong,et al.  Decoding of four movement directions using hybrid NIRS-EEG brain-computer interface , 2014, Front. Hum. Neurosci..

[31]  J. Friedman.  Regularized Discriminant Analysis , 1989 .

[32]  Klaus-Robert Müller,et al.  Co-adaptive calibration to improve BCI efficiency , 2011, Journal of neural engineering.

[33]  Klaus-Robert Müller,et al.  Learning From More Than One Data Source: Data Fusion Techniques for Sensorimotor Rhythm-Based Brain–Computer Interfaces , 2015, Proceedings of the IEEE.

[34]  Chang-Hwan Im,et al.  Evaluation of various mental task combinations for near-infrared spectroscopy-based brain-computer interfaces , 2014, Journal of biomedical optics.

[35]  Stefan Haufe,et al.  The Berlin Brain–Computer Interface: Non-Medical Uses of BCI Technology , 2010, Front. Neurosci..

[36]  J. Wolpaw,et al.  Brain-Computer Interfaces: Principles and Practice , 2012 .

[37]  K. Strimmer,et al.  Statistical Applications in Genetics and Molecular Biology A Shrinkage Approach to Large-Scale Covariance Matrix Estimation and Implications for Functional Genomics , 2011 .

[38]  Larissa C Schudlo,et al.  Dynamic topographical pattern classification of multichannel prefrontal NIRS signals: II. Online differentiation of mental arithmetic and rest , 2014, Journal of neural engineering.

[39]  Febo Cincotti,et al.  Towards Noninvasive Hybrid Brain–Computer Interfaces: Framework, Practice, Clinical Application, and Beyond , 2015, Proceedings of the IEEE.

[40]  Sarah D Power,et al.  Classification of prefrontal activity due to mental arithmetic and music imagery using hidden Markov models and frequency domain near-infrared spectroscopy , 2010, Journal of neural engineering.

[41]  C. Neuper,et al.  Stability of event-related (de-) synchronization during brain–computer interface-relevant mental tasks , 2013, Clinical Neurophysiology.

[42]  Gary H. Glover,et al.  A quantitative comparison of NIRS and fMRI across multiple cognitive tasks , 2011, NeuroImage.

[43]  Keum-Shik Hong,et al.  fNIRS-based brain-computer interfaces: a review , 2015, Front. Hum. Neurosci..

[44]  Terrence J. Sejnowski,et al.  Enhanced detection of artifacts in EEG data using higher-order statistics and independent component analysis , 2007, NeuroImage.

[45]  Hasan Ayaz,et al.  A Methodology for Validating Artifact Removal Techniques for Physiological Signals , 2012, IEEE Transactions on Information Technology in Biomedicine.

[46]  T. Falk,et al.  Taking NIRS-BCIs Outside the Lab: Towards Achieving Robustness Against Environment Noise , 2011, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[47]  A. Villringer,et al.  Simultaneous EEG–fMRI , 2006, Neuroscience & Biobehavioral Reviews.

[48]  Ying Sun,et al.  Asynchronous P300 BCI: SSVEP-based control state detection , 2010, 2010 18th European Signal Processing Conference.

[49]  Dennis J. McFarland,et al.  Brain–computer interfaces for communication and control , 2002, Clinical Neurophysiology.

[50]  Klaus-Robert Müller,et al.  Motor Imagery for Severely Motor-Impaired Patients: Evidence for Brain-Computer Interfacing as Superior Control Solution , 2014, PloS one.

[51]  Motoaki Kawanabe,et al.  Invariant Common Spatial Patterns: Alleviating Nonstationarities in Brain-Computer Interfacing , 2007, NIPS.

[52]  Ann-Christine Ehlis,et al.  Functional near-infrared spectroscopy: A long-term reliable tool for measuring brain activity during verbal fluency , 2008, NeuroImage.

[53]  G. Pfurtscheller,et al.  Self-Paced Operation of an SSVEP-Based Orthosis With and Without an Imagery-Based “Brain Switch:” A Feasibility Study Towards a Hybrid BCI , 2010, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[54]  Jin Uk Kwon,et al.  A Ternary Hybrid EEG-NIRS Brain-Computer Interface for the Classification of Brain Activation Patterns during Mental Arithmetic, Motor Imagery, and Idle State , 2018, Front. Neuroinform..

[55]  Benjamin Blankertz,et al.  Towards a Cure for BCI Illiteracy , 2009, Brain Topography.

[56]  Heidrun Wabnitz,et al.  M3BA: A Mobile, Modular, Multimodal Biosignal Acquisition Architecture for Miniaturized EEG-NIRS-Based Hybrid BCI and Monitoring , 2017, IEEE Transactions on Biomedical Engineering.

[57]  A. Blasi,et al.  Illuminating the developing brain: The past, present and future of functional near infrared spectroscopy , 2010, Neuroscience & Biobehavioral Reviews.

[58]  N. Birbaumer,et al.  fMRI Brain-Computer Interfaces , 2008, IEEE Signal Processing Magazine.

[59]  Han-Jeong Hwang,et al.  Near-infrared spectroscopy (NIRS)-based eyes-closed brain-computer interface (BCI) using prefrontal cortex activation due to mental arithmetic , 2016, Scientific Reports.

[60]  W. De Clercq,et al.  Automatic Removal of Ocular Artifacts in the EEG without an EOG Reference Channel , 2006, Proceedings of the 7th Nordic Signal Processing Symposium - NORSIG 2006.

[61]  A. Shmuel,et al.  Investigation of the initial dip in fMRI at 7 Tesla , 2001, NMR in biomedicine.

[62]  Keum Shik Hong,et al.  Reduction of Delay in Detecting Initial Dips from Functional Near-Infrared Spectroscopy Signals Using Vector-Based Phase Analysis , 2016, Int. J. Neural Syst..

[63]  Klaus-Robert Müller,et al.  Open Access Dataset for EEG+NIRS Single-Trial Classification , 2017, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[64]  Z J Koles,et al.  EEG source localization: implementing the spatio-temporal decomposition approach. , 1998, Electroencephalography and clinical neurophysiology.

[65]  K. Jellinger.  Toward Brain-Computer Interfacing , 2009 .

[66]  Rabab K. Ward,et al.  The Design of a Point-and-Click System by Integrating a Self-Paced Brain–Computer Interface With an Eye-Tracker , 2011, IEEE Journal on Emerging and Selected Topics in Circuits and Systems.

[67]  Stefan Haufe,et al.  Single-trial analysis and classification of ERP components — A tutorial , 2011, NeuroImage.

[68]  Klaus-Robert Müller,et al.  Introduction to machine learning for brain imaging , 2011, NeuroImage.

[69]  Stefan Haufe,et al.  The Berlin Brain-Computer Interface: Progress Beyond Communication and Control , 2016, Front. Neurosci..

[70]  Klaus-Robert Müller,et al.  Evaluation of a Compact Hybrid Brain-Computer Interface System , 2017, BioMed research international.

[71]  Klaus-Robert Müller,et al.  Descriptor : Simultaneous acquisition of EEG and NIRS during cognitive tasks for an open access dataset , 2018 .

[72]  Y. Kim,et al.  Classification of prefrontal and motor cortex signals for three-class fNIRS–BCI , 2015, Neuroscience Letters.

[73]  Tom Chau,et al.  Towards a multimodal brain–computer interface: Combining fNIRS and fTCD measurements to enable higher classification accuracy , 2013, NeuroImage.

[74]  Klaus-Robert Müller,et al.  Enhanced Performance by a Hybrid Nirs–eeg Brain Computer Interface , 2022 .

[75]  Klaus-Robert Müller,et al.  Machine-Learning-Based Coadaptive Calibration for Brain-Computer Interfaces , 2011, Neural Computation.