SST-EmotionNet: Spatial-Spectral-Temporal based Attention 3D Dense Network for EEG Emotion Recognition

Multimedia stimulation of brain activities has not only become an emerging field for intensive research, but also achieves important progress in the electroencephalogram (EEG) emotion classification based on brain activities. However, how to make full use of different EEG features and the discriminative local patterns among the features for different emotions is challenging. Existing models ignore the complementarity among the spatial-spectral-temporal features and discriminative local patterns in all features, which limits the classification ability of the models to a certain extent. In this paper, we propose a novel spatial-spectral-temporal based attention 3D dense network, named SST-EmotionNet, for EEG emotion recognition. The main advantage of the SST-EmotionNet is the simultaneous integration of spatial-spectral-temporal features in a unified network framework. Meanwhile, a 3D attention mechanism is designed to adaptively explore discriminative local patterns. Extensive experiments on two real-world datasets demonstrate that the SST-EmotionNet outperforms the state-of-the-art baselines.

[1]  Wenming Zheng,et al.  Multichannel EEG-Based Emotion Recognition via Group Sparse Canonical Correlation Analysis , 2017, IEEE Transactions on Cognitive and Developmental Systems.

[2]  L. I. Goldfischer.  Autocorrelation Function and Power Spectral Density of Laser-Produced Speckle Patterns , 1965 .

[3]  Youfang Lin,et al.  Refined nonuniform embedding for coupling detection in multivariate time series. , 2020, Physical review. E.

[4]  Bao-Liang Lu,et al.  EEG-Based Emotion Recognition Using Frequency Domain Features and Support Vector Machines , 2011, ICONIP.

[5]  Chunyan Miao,et al.  EEG-Based Emotion Recognition Using Regularized Graph Neural Networks , 2019, IEEE Transactions on Affective Computing.

[6]  Johan A. K. Suykens,et al.  Sparse least squares Support Vector Machine classifiers , 2000, ESANN.

[7]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Zhaoxiang Zhang,et al.  Hierarchical Convolutional Neural Networks for EEG-Based Emotion Recognition , 2017, Cognitive Computation.

[9]  Yuan-Pin Lin,et al.  EEG-Based Emotion Recognition in Music Listening , 2010, IEEE Transactions on Biomedical Engineering.

[10]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[11]  Olga Sourina,et al.  Real-Time Fractal-Based Valence Level Recognition from EEG , 2013, Trans. Comput. Sci..

[12]  Jianmin Jiang,et al.  An Attentional-LSTM for Improved Classification of Brain Activities Evoked by Images , 2019, ACM Multimedia.

[13]  C. L. Philip Chen,et al.  GCB-Net: Graph Convolutional Broad Network and Its Application in Emotion Recognition , 2019, IEEE Transactions on Affective Computing.

[14]  Bao-Liang Lu,et al.  Differential entropy feature for EEG-based emotion classification , 2013, 2013 6th International IEEE/EMBS Conference on Neural Engineering (NER).

[15]  Abeer Al-Nafjan,et al.  Classification of Human Emotions from Electroencephalogram (EEG) Signal using Deep Neural Network , 2017 .

[16]  Terrence J. Sejnowski,et al.  Utilizing Deep Learning Towards Multi-Modal Bio-Sensing and Vision-Based Affective Computing , 2019, IEEE Transactions on Affective Computing.

[17]  Q. M. Jonathan Wu,et al.  EEG-Based Emotion Recognition Using Hierarchical Network With Subnetwork Nodes , 2018, IEEE Transactions on Cognitive and Developmental Systems.

[18]  Jing Wang,et al.  GraphSleepNet: Adaptive Spatial-Temporal Graph Convolutional Networks for Sleep Stage Classification , 2020, IJCAI.

[19]  Charalampos Bratsas,et al.  Toward Emotion Aware Computing: An Integrated Approach Using Multichannel Neurophysiological Recordings and Affective Visual Stimuli , 2010, IEEE Transactions on Information Technology in Biomedicine.

[20]  Yong Peng,et al.  EEG-based emotion classification using deep belief networks , 2014, 2014 IEEE International Conference on Multimedia and Expo (ICME).

[21]  Wenming Zheng,et al.  EEG Emotion Recognition Using Dynamical Graph Convolutional Neural Networks , 2020, IEEE Transactions on Affective Computing.

[22]  Andrzej Cichocki,et al.  EmotionMeter: A Multimodal Framework for Recognizing Human Emotions , 2019, IEEE Transactions on Cybernetics.

[23]  Lei Wang,et al.  A Novel Bi-Hemispheric Discrepancy Model for EEG Emotion Recognition , 2019, IEEE Transactions on Cognitive and Developmental Systems.

[24]  Reda A. El-Khoribi,et al.  Emotion Recognition based on EEG using LSTM Recurrent Neural Network , 2017 .

[25]  Donel M. Martin,et al.  Predicting tDCS treatment outcomes of patients with major depressive disorder using automated EEG classification. , 2017, Journal of affective disorders.

[26]  Bao-Liang Lu,et al.  Classification of Five Emotions from EEG and Eye Movement Signals: Discrimination Ability and Stability over Time , 2019, 2019 9th International IEEE/EMBS Conference on Neural Engineering (NER).

[27]  Amin Janghorbani,et al.  EEG-based emotion recognition using Recurrence Plot analysis and K nearest neighbor classifier , 2013, 2013 20th Iranian Conference on Biomedical Engineering (ICBME).

[28]  Tong Zhang,et al.  A Novel Neural Network Model based on Cerebral Hemispheric Asymmetry for EEG Emotion Recognition , 2018, IJCAI.

[29]  Tao Mei,et al.  Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[30]  Hao Tang,et al.  Emotion Recognition using Multimodal Residual LSTM Network , 2019, ACM Multimedia.

[31]  Jing Wang,et al.  Detecting Causality in Multivariate Time Series via Non-Uniform Embedding , 2019, Entropy.

[32]  G. Knyazev,et al.  Depression and implicit emotion processing: An EEG study , 2017, Neurophysiologie Clinique/Clinical Neurophysiology.

[33]  Zhen Cui,et al.  From Regional to Global Brain: A Novel Hierarchical Spatial-Temporal Neural Network Model for EEG Emotion Recognition , 2019, IEEE Transactions on Affective Computing.

[34]  Adel M. Alimi,et al.  Optimized Echo State Network with Intrinsic Plasticity for EEG-Based Emotion Recognition , 2017, ICONIP.

[35]  Bao-Liang Lu,et al.  Investigating Critical Frequency Bands and Channels for EEG-Based Emotion Recognition with Deep Neural Networks , 2015, IEEE Transactions on Autonomous Mental Development.

[36]  Johan A. K. Suykens,et al.  Least Squares Support Vector Machine Classifiers , 1999, Neural Processing Letters.