Subject-dependent classification for robust idle state detection using multi-modal neuroimaging and data-fusion techniques in BCI

Brain-computer interfaces (BCIs) allow users to control external devices by their intentions. Currently, most BCI systems are synchronous. They rely on cues or tasks to which a subject has to react. In order to design an asynchronous BCI one needs to be able to robustly detect an idle class. In this study, we examine whether multi-modal neuroimaging, based on simultaneous EEG and near-infrared spectroscopy (NIRS) measurements, can assist in the robust detection of the idle class within a sensory motor rhythm-based BCI paradigm. We propose two types of subject-dependent classification strategies to combine the information of both modalities. Our results demonstrate that not only idle-state decoding can be significantly improved by exploiting the complementary information of multi-modal recordings, but also it is possible to minimize the delay of the system, caused by the slow inherent hemodynamic response of the NIRS signal. HighlightsEEG-NIRS measurements can robustly detect the idle class.An activation function to reduce hemodynamic response delays in NIRS is proposed.A novel hybrid classification strategy combining EEG-NIRS classifiers is proposed.Superior performance by complementary information of multimodal recordings.

[1]  Ilya Narsky,et al.  Reducing Multiclass to Binary , 2013 .

[2]  Terrence J. Sejnowski,et al.  Toward Brain-Computer Interfacing (Neural Information Processing) , 2007 .

[3]  Yoram Singer,et al.  Reducing Multiclass to Binary: A Unifying Approach for Margin Classifiers , 2000, J. Mach. Learn. Res..

[4]  Klaus-Robert Müller,et al.  Integration of Multivariate Data Streams With Bandpower Signals , 2013, IEEE Transactions on Multimedia.

[5]  Keinosuke Fukunaga,et al.  Introduction to statistical pattern recognition (2nd ed.) , 1990 .

[6]  R. Turner,et al.  Dynamic magnetic resonance imaging of human brain activity during primary sensory stimulation. , 1992, Proceedings of the National Academy of Sciences of the United States of America.

[7]  Cuntai Guan,et al.  Temporal classification of multichannel near-infrared spectroscopy signals of motor imagery for developing a brain–computer interface , 2007, NeuroImage.

[8]  Andrzej Cichocki,et al.  Bimodal BCI Using Simultaneously NIRS and EEG , 2014, IEEE Transactions on Biomedical Engineering.

[9]  D. Tucker,et al.  EEG coherency. I: Statistics, reference electrode, volume conduction, Laplacians, cortical imaging, and interpretation at multiple scales. , 1997, Electroencephalography and clinical neurophysiology.

[10]  Heung-Il Suk,et al.  A Novel Bayesian Framework for Discriminative Feature Extraction in Brain-Computer Interfaces , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[11]  Dimitri Van De Ville,et al.  Brain decoding: Opportunities and challenges for pattern recognition , 2012, Pattern Recognit..

[12]  Klaus-Robert Müller,et al.  Classifying Single Trial EEG: Towards Brain Computer Interfacing , 2001, NIPS.

[13]  Heung-Il Suk,et al.  Person authentication from neural activity of face-specific visual self-representation , 2013, Pattern Recognit..

[14]  Benjamin Blankertz,et al.  Towards a Cure for BCI Illiteracy , 2009, Brain Topography.

[15]  Klaus-Robert Müller,et al.  Boosting bit rates in noninvasive EEG single-trial classifications by feature combination and multiclass paradigms , 2004, IEEE Transactions on Biomedical Engineering.

[16]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[17]  Klaus-Robert Müller,et al.  Temporal kernel CCA and its application in multimodal neuronal data analysis , 2010, Machine Learning.

[18]  Maureen Clerc,et al.  An analysis of performance evaluation for motor-imagery based BCI , 2013, Journal of neural engineering.

[19]  Rajesh P. N. Rao.  Brain-Computer Interfacing: An Introduction , 2010 .

[20]  Klaus-Robert Müller,et al.  Enhanced Performance by a Hybrid Nirs–eeg Brain Computer Interface , 2022 .

[21]  Klaus-Robert Müller,et al.  Machine-Learning-Based Coadaptive Calibration for Brain-Computer Interfaces , 2011, Neural Computation.

[22]  M. Nuttin,et al.  A brain-actuated wheelchair: Asynchronous and non-invasive Brain–computer interfaces for continuous control of robots , 2008, Clinical Neurophysiology.

[23]  Reinhold Scherer,et al.  Direct Brain-Computer Communication , 2004 .

[24]  G. Pfurtscheller,et al.  Graz-BCI: state of the art and clinical applications , 2003, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[25]  Gary E. Birch,et al.  A brain-controlled switch for asynchronous control applications , 2000, IEEE Trans. Biomed. Eng..

[26]  F. Meinecke,et al.  Analysis of Multimodal Neuroimaging Data , 2011, IEEE Reviews in Biomedical Engineering.

[27]  N. Birbaumer,et al.  Brain-computer communication: self-regulation of slow cortical potentials for verbal communication. , 2001, Archives of physical medicine and rehabilitation.

[28]  Keum-Shik Hong,et al.  Decoding of four movement directions using hybrid NIRS-EEG brain-computer interface , 2014, Front. Hum. Neurosci..

[29]  Tom Chau,et al.  Decoding subjective preference from single-trial near-infrared spectroscopy signals , 2009, Journal of neural engineering.

[30]  Christoph M. Michel,et al.  Decoding stimulus-related information from single-trial EEG responses based on voltage topographies , 2012, Pattern Recognit..

[31]  K.-R. Muller,et al.  Optimizing Spatial filters for Robust EEG Single-Trial Analysis , 2008, IEEE Signal Processing Magazine.

[32]  Cuntai Guan,et al.  Mutual information-based selection of optimal spatial-temporal patterns for single-trial EEG-based BCIs , 2012, Pattern Recognit..

[33]  J. Mourino,et al.  Asynchronous BCI and local neural classifiers: an overview of the adaptive brain interface project , 2003, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[34]  Keinosuke Fukunaga,et al.  Introduction to Statistical Pattern Recognition , 1972 .

[35]  Siamac Fazli,et al.  Improving the Performance of Brain-Computer Interface Using Multi-modal Neuroimaging , 2013, 2013 2nd IAPR Asian Conference on Pattern Recognition.

[36]  Brendan Z. Allison,et al.  The Hybrid BCI , 2010, Frontiers in Neuroscience.

[37]  Siamac Fazli,et al.  Brain Computer Interfacing: A Multi-Modal Perspective , 2013, J. Comput. Sci. Eng..

[38]  T. Chau,et al.  Towards a system-paced near-infrared spectroscopy brain–computer interface: differentiating prefrontal activity due to mental arithmetic and mental singing from the no-control state , 2011, Journal of neural engineering.

[39]  E. DeYoe,et al.  Analysis and use of FMRI response delays , 2001, Human brain mapping.

[40]  A. Eke,et al.  The modified Beer–Lambert law revisited , 2006, Physics in medicine and biology.

[41]  R. Fisher.  THE USE OF MULTIPLE MEASUREMENTS IN TAXONOMIC PROBLEMS , 1936 .

[42]  Yijun Wang,et al.  An Algorithm for Idle-State Detection in Motor-Imagery-Based Brain-Computer Interface , 2007, Comput. Intell. Neurosci..