Classification of Motor Imagery EEG Based on Time-Domain and Frequency-Domain Dual-Stream Convolutional Neural Network

Abstract Background and objective An important task of the brain-computer interface (BCI) of motor imagery is to extract effective time-domain features, frequency-domain features or time-frequency domain features from the raw electroencephalogram (EEG) signals for classification of motor imagery. However, choosing an appropriate method to combine time domain and frequency domain features to improve the performance of motor imagery recognition is still a research hotspot. Methods In order to fully extract and utilize the time-domain and frequency-domain features of EEG in classification tasks, this paper proposed a novel dual-stream convolutional neural network (DCNN), which can use time domain signal and frequency domain signal as the inputs, and the extracted time-domain features and frequency-domain features are fused by linear weighting for classification training. Furthermore, the weight can be learned by the DCNN automatically. Results The experiments based on BCI competition II dataset III and BCI competition IV dataset 2a showed that the model proposed by this study has better performance than other conventional methods. The model used time-frequency signal as the inputs had better performance than the model only used time-domain signals or frequency-domain signals. The accuracy of classification was improved for each subject compared with the models only used one signals as the inputs. Conclusions Further analysis shown that the fusion weight of different subject is specifically, adjusting the weight coefficient automatically is helpful to improve the classification accuracy.

[1]  Muhammad Ghulam,et al.  Deep Learning for EEG motor imagery classification based on multi-layer CNNs feature fusion , 2019, Future Gener. Comput. Syst..

[2]  Keiron O'Shea,et al.  An Introduction to Convolutional Neural Networks , 2015, ArXiv.

[3]  Ugur Halici,et al.  A novel deep learning approach for classification of EEG motor imagery signals , 2017, Journal of neural engineering.

[4]  Lining Sun,et al.  A Multi-Branch 3D Convolutional Neural Network for EEG-Based Motor Imagery Classification , 2019, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[5]  Andrew Zisserman,et al.  Two-Stream Convolutional Networks for Action Recognition in Videos , 2014, NIPS.

[6]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[7]  M Congedo,et al.  A review of classification algorithms for EEG-based brain–computer interfaces: a 10 year update , 2018, Journal of neural engineering.

[8]  Klaus-Robert Müller,et al.  The BCI competition 2003: progress and perspectives in detection and discrimination of EEG single trials , 2004, IEEE Transactions on Biomedical Engineering.

[9]  Paul R. Carney,et al.  Reading EEGs: A Practical Approach , 2009 .

[10]  Brent Lance,et al.  EEGNet: a compact convolutional neural network for EEG-based brain–computer interfaces , 2016, Journal of neural engineering.

[11]  Ah Chung Tsoi,et al.  Face recognition: a convolutional neural-network approach , 1997, IEEE Trans. Neural Networks.

[12]  Gao Xiaorong,et al.  Outcome of the BCI-competition 2003 on the Graz data set , 2003 .

[13]  Swati Aggarwal,et al.  Signal processing techniques for motor imagery brain computer interface: A review , 2019, Array.

[14]  Cuntai Guan,et al.  On the use of convolutional neural networks and augmented CSP features for multi-class motor imagery of EEG signals classification , 2015, 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[15]  Wolfram Burgard,et al.  Deep learning with convolutional neural networks for EEG decoding and visualization , 2017, Human brain mapping.

[16]  R. Leeb,et al.  BCI Competition 2008 { Graz data set B , 2008 .

[17]  Gert Pfurtscheller,et al.  EEG event-related desynchronization (ERD) and synchronization (ERS) , 1997 .

[18]  Sepp Hochreiter,et al.  Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) , 2015, ICLR.

[19]  Han-Jeong Hwang,et al.  Neurofeedback-based motor imagery training for brain–computer interface (BCI) , 2009, Journal of Neuroscience Methods.

[20]  Cuntai Guan,et al.  Filter Bank Common Spatial Pattern Algorithm on BCI Competition IV Datasets 2a and 2b , 2012, Front. Neurosci..

[21]  Abdulkadir Sengur,et al.  Convolutional Neural Network Based Approach Towards Motor Imagery Tasks EEG Signals Classification , 2019, IEEE Sensors Journal.

[22]  Chunfang Liu,et al.  Multimodal Classification with Deep Convolutional-Recurrent Neural Networks for Electroencephalography , 2017, ICONIP.

[23]  Muhammad Ghulam,et al.  Multi-CNN Feature Fusion for Efficient EEG Classification , 2020, 2020 IEEE International Conference on Multimedia & Expo Workshops (ICMEW).

[24]  A. Akrami,et al.  EEG-Based Mental Task Classification: Linear and Nonlinear Classification of Movement Imagery , 2005, 2005 IEEE Engineering in Medicine and Biology 27th Annual Conference.

[25]  Swati Vaid,et al.  EEG Signal Analysis for BCI Interface: A Review , 2015, 2015 Fifth International Conference on Advanced Computing & Communication Technologies.

[26]  Shuicheng Yan,et al.  Learning Temporal Information for Brain-Computer Interface Using Convolutional Neural Networks , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[27]  Gloria M. Díaz,et al.  Deep Convolutional Neural Networks and Power Spectral Density Features for Motor Imagery Classification of EEG Signals , 2018, HCI.

[28]  Yan Wu,et al.  Convolutional deep belief networks for feature extraction of EEG signal , 2014, 2014 International Joint Conference on Neural Networks (IJCNN).

[29]  Aiguo Song,et al.  Wavelet Transform Time-Frequency Image and Convolutional Network-Based Motor Imagery EEG Classification , 2019, IEEE Access.