Convolutional Neural Network for Hybrid fNIRS-EEG Mental Workload Classification

The classification of workload memory tasks based on fNIRS and EEG signals requires solving high-dimensional pattern classification problems with a relatively small number of training patterns. In the use of conventional machine learning algorithms, feature selection is a fundamental difficulty given the large number of possible features and the small amount of available data. In this study, we bypass the challenges of feature selection and investigate the use of Convolutional Neural Networks (CNNs) for classifying workload memory tasks. CNNs are well suited for learning from the raw data without any a priori feature selection. CNNs take as input two-dimensional images, which differ in structure from the neural time series obtained on the scalp surface using EEG and fNIRS. Therefore, both the existing CNN architectures and fNIRS-EEG input must be adapted to allow fNIRS-EEG input to a CNN. In this work, we describe this adaptation, evaluate the performance of CNN classification of mental workload tasks. This study makes use of an open-source meta-dataset collected at the Technische Universitat Berlin; including simultaneous EEG and fNIRS recordings of 26 healthy participants during n-back tests. A CNN with three convolution layers and two fully connected layers is adapted to suit the given dataset. ReLU and ELU activation functions are employed to take advantage of their better dampening property in the vanishing gradient problem, fast convergence, and higher accuracy. The results achieved with the two activation functions are compared to select the best performing function. The proposed CNN approach achieves a considerable average improvement relative to conventional methods such as Support Vector Machines. The results across differences in time window length, activation functions, and other hyperparameters are benchmarked for each task. The best result is obtained with a three-second window and the ELU activation function, for which the CNN yields 89% correct classification, while the SVM achieves only 82% correct classification.

[1]  Klaus-Robert Müller,et al.  Descriptor : Simultaneous acquisition of EEG and NIRS during cognitive tasks for an open access dataset , 2018 .

[2]  Tzyy-Ping Jung,et al.  EEG-based prediction of driver's cognitive performance by deep convolutional neural network , 2016, Signal Process. Image Commun..

[3]  Siamac Fazli,et al.  Hybrid brain-computer interface based on EEG and NIRS modalities , 2014, 2014 International Winter Workshop on Brain-Computer Interface (BCI).

[4]  Marjan Saadati,et al.  Multiple transmitter localization using clustering by likelihood of transmitter proximity , 2017, 2017 51st Asilomar Conference on Signals, Systems, and Computers.

[5]  Y. Kim,et al.  Classification of prefrontal and motor cortex signals for three-class fNIRS–BCI , 2015, Neuroscience Letters.

[6]  Klaus-Robert Müller,et al.  Enhanced Performance by a Hybrid Nirs–eeg Brain Computer Interface , 2022 .

[7]  Jürgen Schmidhuber,et al.  Deep learning in neural networks: An overview , 2014, Neural Networks.

[8]  Wolfram Burgard,et al.  Deep learning with convolutional neural networks for EEG decoding and visualization , 2017, Human brain mapping.

[9]  Lu Wang,et al.  A hybrid brain-computer interface combining the EEG and NIRS , 2012, 2012 IEEE International Conference on Virtual Environments Human-Computer Interfaces and Measurement Systems (VECIMS) Proceedings.

[10]  Joelle Pineau,et al.  Learning Robust Features using Deep Learning for Automatic Seizure Detection , 2016, MLHC.

[11]  Filippo Zappasodi,et al.  Exploiting neurovascular coupling: a Bayesian sequential Monte Carlo approach applied to simulated EEG fNIRS data. , 2017, Journal of neural engineering.

[12]  Jerzy Jarmasz,et al.  Object-based attention and cognitive tunneling. , 2005, Journal of experimental psychology. Applied.

[13]  Mohammed Yeasin,et al.  Learning Representations from EEG with Deep Recurrent-Convolutional Neural Networks , 2015, ICLR.

[14]  Marc Garbey,et al.  Measuring Mental Workload with EEG+fNIRS , 2017, Front. Hum. Neurosci..

[15]  S.S. Mortazavi,et al.  Optimal reactive power compensation in a deregulated distribution network , 2009, 2009 44th International Universities Power Engineering Conference (UPEC).

[16]  Filippo Zappasodi,et al.  Simultaneous functional near-infrared spectroscopy and electroencephalography for monitoring of human brain activity and oxygenation: a review , 2017, Neurophotonics.

[17]  Hasan Ayaz,et al.  Multisubject “Learning” for Mental Workload Classification Using Concurrent EEG, fNIRS, and Physiological Measures , 2017, Front. Hum. Neurosci..

[18]  Pasin Israsena,et al.  EEG-Based Emotion Recognition Using Deep Learning Network with Principal Component Based Covariate Shift Adaptation , 2014, TheScientificWorldJournal.

[19]  Ji-Woong Choi,et al.  Convolutional neural network for high-accuracy functional near-infrared spectroscopy in a brain–computer interface: three-class classification of rest, right-, and left-hand motor execution , 2017, Neurophotonics.

[20]  Keum-Shik Hong,et al.  Decoding of four movement directions using hybrid NIRS-EEG brain-computer interface , 2014, Front. Hum. Neurosci..

[21]  G. Pfurtscheller,et al.  Functional brain imaging based on ERD/ERS , 2001, Vision Research.

[22]  G. Pfurtscheller,et al.  Event-related dynamics of cortical rhythms: frequency-specific features and functional correlates. , 2001, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[23]  Hasan Ayaz,et al.  Comparison of Machine Learning Approaches for Motor Imagery Based Optical Brain Computer Interface , 2018, Advances in Neuroergonomics and Cognitive Engineering.