SAE+LSTM: A New Framework for Emotion Recognition From Multi-Channel EEG

EEG-based automatic emotion recognition can help brain-inspired robots in improving their interactions with humans. This paper presents a novel framework for emotion recognition using multi-channel electroencephalogram (EEG). The framework consists of a linear EEG mixing model and an emotion timing model. Our proposed framework considerably decomposes the EEG source signals from the collected EEG signals and improves classification accuracy by using the context correlations of the EEG feature sequences. Specially, Stack AutoEncoder (SAE) is used to build and solve the linear EEG mixing model and the emotion timing model is based on the Long Short-Term Memory Recurrent Neural Network (LSTM-RNN). The framework was implemented on the DEAP dataset for an emotion recognition experiment, where the mean accuracy of emotion recognition achieved 81.10% in valence and 74.38% in arousal, and the effectiveness of our framework was verified. Our framework exhibited a better performance in emotion recognition using multi-channel EEG than the compared conventional approaches in the experiments.

[1]  Oluwarotimi Williams Samuel,et al.  Towards Efficient Decoding of Multiple Classes of Motor Imagery Limb Movements Based on EEG Spectral and Time Domain Descriptors , 2017, Journal of Medical Systems.

[2]  Xiang Li,et al.  Emotion recognition from multi-channel EEG data through Convolutional Recurrent Neural Network , 2016, 2016 IEEE International Conference on Bioinformatics and Biomedicine (BIBM).

[3]  Tiago H. Falk,et al.  Relevance vector classifier decision fusion and EEG graph-theoretic features for automatic affective state characterization , 2016, Neurocomputing.

[4]  Mohammad Soleymani,et al.  Continuous emotion detection using EEG signals and facial expressions , 2014, 2014 IEEE International Conference on Multimedia and Expo (ICME).

[5]  Huimin Lu,et al.  Facial Emotion Recognition Based on Biorthogonal Wavelet Entropy, Fuzzy Support Vector Machine, and Stratified Cross Validation , 2016, IEEE Access.

[6]  Rabab Kreidieh Ward,et al.  Semi-supervised Stacked Label Consistent Autoencoder for Reconstruction and Analysis of Biomedical Signals , 2017, IEEE Transactions on Biomedical Engineering.

[7]  Miguel Arevalillo-Herráez,et al.  Fusing highly dimensional energy and connectivity features to identify affective states from EEG signals , 2017, Neurocomputing.

[8]  Yongzhao Zhan,et al.  Learning Salient Features for Speech Emotion Recognition Using Convolutional Neural Networks , 2014, IEEE Transactions on Multimedia.

[9]  Geoffrey E. Hinton,et al.  Reducing the Dimensionality of Data with Neural Networks , 2006, Science.

[10]  Yuan-Pin Lin,et al.  EEG-Based Emotion Recognition in Music Listening , 2010, IEEE Transactions on Biomedical Engineering.

[11]  Kuldip K. Paliwal,et al.  Bidirectional recurrent neural networks , 1997, IEEE Trans. Signal Process..

[12]  P. Lang.  The emotion probe. Studies of motivation and attention. , 1995, The American psychologist.

[13]  Mohamed Hamid,et al.  Applying ICA in EEG: Choice of the Window Length and of the Decorrelation Method , 2012, BIOSTEC.

[14]  Nitin Kumar,et al.  Bispectral Analysis of EEG for Emotion Recognition , 2015, IHCI.

[15]  Andrew W. Senior,et al.  Long short-term memory recurrent neural network architectures for large scale acoustic modeling , 2014, INTERSPEECH.

[16]  Oluwarotimi Williams Samuel,et al.  A motion-classification strategy based on sEMG-EEG signal combination for upper-limb amputees , 2017, Journal of NeuroEngineering and Rehabilitation.

[17]  Yoshua Bengio,et al.  Learning long-term dependencies with gradient descent is difficult , 1994, IEEE Trans. Neural Networks.

[18]  Javad Frounchi,et al.  Wavelet-based emotion recognition system using EEG signal , 2017, Neural Computing and Applications.

[19]  Zhenqi Li,et al.  Emotion Recognition from EEG Using RASM and LSTM , 2017, ICIMCS.

[20]  Sanjiv Kumar,et al.  On the Convergence of Adam and Beyond , 2018 .

[21]  Wei Liu,et al.  Emotion Recognition Using Multimodal Deep Learning , 2016, ICONIP.

[22]  Wei Zhang,et al.  Cross-Subject EEG Feature Selection for Emotion Recognition Using Transfer Recursive Feature Elimination , 2017, Front. Neurorobot..

[23]  You Rongyi,et al.  Blind signal separation of multi-channel EEG , 2004 .

[24]  Hiroshi Ishiguro,et al.  EEG theta and Mu oscillations during perception of human and robot actions , 2013, Front. Neurorobot..

[25]  Bin Hu,et al.  Feature-level fusion of multimodal physiological signals for emotion recognition , 2015, 2015 IEEE International Conference on Bioinformatics and Biomedicine (BIBM).

[26]  Clemens Brunner,et al.  Spatial filtering and selection of optimized components in four class motor imagery EEG data using independent components analysis , 2007, Pattern Recognit. Lett..

[27]  W Singer,et al.  Visual feature integration and the temporal correlation hypothesis. , 1995, Annual review of neuroscience.

[28]  Hiie Hinrikus,et al.  Effect of negative and positive emotions on EEG spectral asymmetry , 2015, 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[29]  Bin Hu,et al.  Electroencephalogram-based emotion assessment system using ontology and data mining techniques , 2015, Appl. Soft Comput..

[30]  Jiahui Wang,et al.  Identifying valence and arousal levels via connectivity between EEG channels , 2015, 2015 International Conference on Affective Computing and Intelligent Interaction (ACII).

[31]  Jeremy B. Caplan,et al.  A better oscillation detection method robustly extracts EEG rhythms across brain state changes: The human alpha rhythm as a test case , 2011, NeuroImage.

[32]  Zhenqi Li,et al.  A Review of Emotion Recognition Using Physiological Signals , 2018, Sensors.

[33]  P. Rapp,et al.  Time domain measures of inter-channel EEG correlations: a comparison of linear, nonparametric and nonlinear measures , 2013, Cognitive Neurodynamics.

[34]  Thierry Pun,et al.  DEAP: A Database for Emotion Analysis ;Using Physiological Signals , 2012, IEEE Transactions on Affective Computing.

[35]  Rafael A. Calvo,et al.  Detecting Naturalistic Expressions of Nonbasic Affect Using Physiological Signals , 2012, IEEE Transactions on Affective Computing.

[36]  Bao-Liang Lu,et al.  Identifying Stable Patterns over Time for Emotion Recognition from EEG , 2016, IEEE Transactions on Affective Computing.

[37]  M. Bradley,et al.  Large-scale neural correlates of affective picture processing. , 2002, Psychophysiology.

[38]  E. Phelps.  Human emotion and memory: interactions of the amygdala and hippocampal complex , 2004, Current Opinion in Neurobiology.

[39]  Ernst Fernando Lopes Da Silva Niedermeyer,et al.  Electroencephalography, basic principles, clinical applications, and related fields , 1982 .

[40]  Yoonsuck Choe,et al.  Predictable internal brain dynamics in EEG and its relation to conscious states , 2014, Front. Neurorobot..

[41]  Mi Li,et al.  Emotion recognition from multichannel EEG signals using K-nearest neighbor classification , 2018, Technology and health care : official journal of the European Society for Engineering and Medicine.

[42]  Kai Keng Ang,et al.  ERNN: A Biologically Inspired Feedforward Neural Network to Discriminate Emotion From EEG Signal , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[43]  F. L. D. Silva,et al.  EEG signal processing , 2000, Clinical Neurophysiology.

[44]  Patrice Simardy,et al.  Learning Long-Term Dependencies with , 2007 .

[45]  Mohammad Soleymani,et al.  Analysis of EEG Signals and Facial Expressions for Continuous Emotion Detection , 2016, IEEE Transactions on Affective Computing.

[46]  S. A. Hosseini,et al.  Emotion recognition method using entropy analysis of EEG signals , 2011 .