Motor Imagery Classification based on CNN-GRU Network with Spatio-Temporal Feature Representation

Recently, various deep neural networks have been applied to classify electroencephalogram (EEG) signal. EEG is a brain signal that can be acquired in a non-invasive way and has a high temporal resolution. It can be used to decode the intention of users. As the EEG signal has a high dimension of feature space, appropriate feature extraction methods are needed to improve classification performance. In this study, we obtained spatio-temporal feature representation and classified them with the combined convolutional neural networks (CNN)-gated recurrent unit (GRU) model. To this end, we obtained covariance matrices in each different temporal band and then concatenated them on the temporal axis to obtain a final spatio-temporal feature representation. In the classification model, CNN is responsible for spatial feature extraction and GRU is responsible for temporal feature extraction. Classification performance was improved by distinguishing spatial data processing and temporal data processing. The average accuracy of the proposed model was 77.70% (± 15.39) for the BCI competition IV 2a data set. The proposed method outperformed all other methods compared as a baseline method.

[1]  Klaus-Robert Müller,et al.  Subject-independent mental state classification in single trials , 2009, Neural Networks.

[2]  Bin He,et al.  Brain–Computer Interfaces Using Sensorimotor Rhythms: Current State and Future Perspectives , 2014, IEEE Transactions on Biomedical Engineering.

[3]  G. Pfurtscheller,et al.  Optimal spatial filtering of single trial EEG during imagined hand movement. , 2000, IEEE transactions on rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society.

[4]  Damien Coyle,et al.  Neural network based auto association and time-series prediction for biosignal processing in brain-computer interfaces , 2009, IEEE Computational Intelligence Magazine.

[5]  John Williamson,et al.  EEG dataset and OpenBMI toolbox for three BCI paradigms: an investigation into BCI illiteracy , 2019, GigaScience.

[6]  Gert Pfurtscheller,et al.  Motor imagery and direct brain-computer communication , 2001, Proc. IEEE.

[7]  Hubert Cecotti,et al.  Convolutional Neural Networks for P300 Detection with Application to Brain-Computer Interfaces , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[8]  John Williamson,et al.  A High Performance Spelling System based on EEG-EOG Signals With Visual Feedback , 2018, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[9]  Klaus-Robert Müller,et al.  Spatio-spectral filters for improving the classification of single trial EEG , 2005, IEEE Transactions on Biomedical Engineering.

[10]  Heung-Il Suk,et al.  Subject and class specific frequency bands selection for multiclass motor imagery classification , 2011, Int. J. Imaging Syst. Technol..

[11]  Seong-Whan Lee,et al.  Subject-Independent Brain–Computer Interfaces Based on Deep Convolutional Neural Networks , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[12]  Lawrence D. Jackel,et al.  Handwritten Digit Recognition with a Back-Propagation Network , 1989, NIPS.

[13]  Mohammed Yeasin,et al.  Learning Representations from EEG with Deep Recurrent-Convolutional Neural Networks , 2015, ICLR.

[14]  Cuntai Guan,et al.  Filter Bank Common Spatial Pattern Algorithm on BCI Competition IV Datasets 2a and 2b , 2012, Front. Neurosci..

[15]  Cuntai Guan,et al.  Mutual information-based selection of optimal spatial-temporal patterns for single-trial EEG-based BCIs , 2012, Pattern Recognit..

[16]  K.-R. Muller,et al.  Optimizing Spatial filters for Robust EEG Single-Trial Analysis , 2008, IEEE Signal Processing Magazine.

[17]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[18]  Ad Aertsen,et al.  Review of the BCI Competition IV , 2012, Front. Neurosci..

[19]  Ran Manor,et al.  Convolutional Neural Network for Multi-Category Rapid Serial Visual Presentation BCI , 2015, Front. Comput. Neurosci..

[20]  Sebastian Stober,et al.  Using Convolutional Neural Networks to Recognize Rhythm Stimuli from Electroencephalography Recordings , 2014, NIPS.

[21]  Hyoung Joong Kim,et al.  A High-Security EEG-Based Login System with RSVP Stimuli and Dry Electrodes , 2016, IEEE Transactions on Information Forensics and Security.

[22]  Klaus-Robert Müller,et al.  Motion-Based Rapid Serial Visual Presentation for Gaze-Independent Brain-Computer Interfaces , 2018, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[23]  Seong-Whan Lee,et al.  Spatio-Spectral Feature Representation for Motor Imagery Classification Using Convolutional Neural Networks , 2021, IEEE Transactions on Neural Networks and Learning Systems.

[24]  Geoffrey E. Hinton,et al.  Rectified Linear Units Improve Restricted Boltzmann Machines , 2010, ICML.

[25]  Cuntai Guan,et al.  Regularizing Common Spatial Patterns to Improve BCI Designs: Unified Theory and New Algorithms , 2011, IEEE Transactions on Biomedical Engineering.

[26]  Klaus-Robert Müller,et al.  Interpretable deep neural networks for single-trial EEG classification , 2016, Journal of Neuroscience Methods.

[27]  Christian Jutten,et al.  Multiclass Brain–Computer Interface Classification by Riemannian Geometry , 2012, IEEE Transactions on Biomedical Engineering.

[28]  Clemens Brunner,et al.  Mu rhythm (de)synchronization and EEG single-trial classification of different motor imagery tasks , 2006, NeuroImage.

[29]  Wolfram Burgard,et al.  Deep learning with convolutional neural networks for EEG decoding and visualization , 2017, Human brain mapping.

[30]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[31]  Girijesh Prasad,et al.  Neural time-series prediction preprocessing meets common spatial patterns in a brain-computer interface , 2008, 2008 30th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[32]  Shuicheng Yan,et al.  Learning Temporal Information for Brain-Computer Interface Using Convolutional Neural Networks , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[33]  Klaus-Robert Müller,et al.  Towards Zero Training for Brain-Computer Interfacing , 2008, PloS one.