Multi-channel EEG-based emotion recognition via a multi-level features guided capsule network

In recent years, deep learning (DL) techniques, and in particular convolutional neural networks (CNNs), have shown great potential in electroencephalograph (EEG)-based emotion recognition. However, existing CNN-based EEG emotion recognition methods usually require a relatively complex stage of feature pre-extraction. More importantly, the CNNs cannot well characterize the intrinsic relationship among the different channels of EEG signals, which is essentially a crucial clue for the recognition of emotion. In this paper, we propose an effective multi-level features guided capsule network (MLF-CapsNet) for multi-channel EEG-based emotion recognition to overcome these issues. The MLF-CapsNet is an end-to-end framework, which can simultaneously extract features from the raw EEG signals and determine the emotional states. Compared with original CapsNet, it incorporates multi-level feature maps learned by different layers in forming the primary capsules so that the capability of feature representation can be enhanced. In addition, it uses a bottleneck layer to reduce the amount of parameters and accelerate the speed of calculation. Our method achieves the average accuracy of 97.97%, 98.31% and 98.32% on valence, arousal and dominance of DEAP dataset, respectively, and 94.59%, 95.26% and 95.13% on valence, arousal and dominance of DREAMER dataset, respectively. These results show that our method exhibits higher accuracy than the state-of-the-art methods.

[1]  Tardi Tjahjadi,et al.  A Mutual Information Based Adaptive Windowing of Informative EEG for Emotion Recognition , 2020, IEEE Transactions on Affective Computing.

[2]  Terrence J. Sejnowski,et al.  Utilizing Deep Learning Towards Multi-Modal Bio-Sensing and Vision-Based Affective Computing , 2019, IEEE Transactions on Affective Computing.

[3]  Xiaowei Chen,et al.  Continuous Convolutional Neural Network with 3D Input for EEG-Based Emotion Recognition , 2018, ICONIP.

[4]  Dongmei Jiang,et al.  Emotion recognition from spatiotemporal EEG representations with hybrid convolutional recurrent neural networks via wearable multi-channel headset , 2020, Comput. Commun..

[5]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  Gene H. Golub,et al.  Generalized cross-validation as a method for choosing a good ridge parameter , 1979, Milestones in Matrix Computation.

[7]  Kristen A. Lindquist,et al.  A functional architecture of the human brain: emerging insights from the science of emotion , 2012, Trends in Cognitive Sciences.

[8]  Ming Qiu,et al.  Emotion Recognition from Multi-Channel EEG through Parallel Convolutional Recurrent Neural Network , 2018, 2018 International Joint Conference on Neural Networks (IJCNN).

[9]  Martín Abadi,et al.  TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems , 2016, ArXiv.

[10]  Jiahui Pan,et al.  Speech emotion recognition using fusion of three multi-task learning-based classifiers: HSF-DNN, MS-CNN and LLD-RNN , 2020, Speech Commun..

[11]  Ehsan Lotfi,et al.  Practical emotional neural networks , 2014, Neural Networks.

[12]  Yuan-Pin Lin,et al.  EEG-Based Emotion Recognition in Music Listening , 2010, IEEE Transactions on Biomedical Engineering.

[13]  Dimitrios Hatzinakos,et al.  ECG Pattern Analysis for Emotion Detection , 2012, IEEE Transactions on Affective Computing.

[14]  Wei Liu,et al.  Multimodal Emotion Recognition Using Deep Canonical Correlation Analysis , 2019, ArXiv.

[15]  Xun Chen,et al.  Robust Multichannel EEG Compressed Sensing in the Presence of Mixed Noise , 2019, IEEE Sensors Journal.

[16]  Leontios J. Hadjileontiadis,et al.  Emotion Recognition From EEG Using Higher Order Crossings , 2010, IEEE Transactions on Information Technology in Biomedicine.

[17]  Samit Bhattacharya,et al.  Using Deep and Convolutional Neural Networks for Accurate Emotion Classification on DEAP Dataset , 2017, AAAI.

[18]  Hatice Gunes,et al.  Bi-modal emotion recognition from expressive face and body gestures , 2007, J. Netw. Comput. Appl..

[19]  Johan A. K. Suykens,et al.  Least Squares Support Vector Machine Classifiers , 1999, Neural Processing Letters.

[20]  Lina Yao,et al.  EEG-based Intention Recognition from Spatio-Temporal Representations via Cascade and Parallel Convolutional Recurrent Neural Networks , 2017, ArXiv.

[21]  Thierry Pun,et al.  DEAP: A Database for Emotion Analysis ;Using Physiological Signals , 2012, IEEE Transactions on Affective Computing.

[22]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[23]  Seong-Eun Moon,et al.  Convolutional Neural Network Approach for Eeg-Based Emotion Recognition Using Brain Connectivity and its Spatial Information , 2018, 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[24]  S. Hsieh,et al.  Classifying Different Emotional States by Means of EEG-Based Functional Connectivity Patterns , 2014, PloS one.

[25]  Liang Dong,et al.  Recognition of Emotions Using Multichannel EEG Data and DBN-GC-Based Ensemble Deep Learning Framework , 2018, Comput. Intell. Neurosci..

[26]  Areej Al-Wabil,et al.  Review and Classification of Emotion Recognition Based on EEG Brain-Computer Interface System Research: A Systematic Review , 2017 .

[27]  Yanli Li,et al.  A Channel-Fused Dense Convolutional Network for EEG-Based Emotion Recognition , 2020, IEEE Transactions on Cognitive and Developmental Systems.

[28]  Jihao Yin,et al.  Hyperspectral Image Classification Using CapsNet With Well-Initialized Shallow Layers , 2019, IEEE Geoscience and Remote Sensing Letters.

[29]  Bao-Liang Lu,et al.  Emotional state classification from EEG data using machine learning approach , 2014, Neurocomputing.

[30]  Geoffrey E. Hinton,et al.  Dynamic Routing Between Capsules , 2017, NIPS.

[31]  Olga Sourina,et al.  Real-Time Fractal-Based Valence Level Recognition from EEG , 2013, Trans. Comput. Sci..

[32]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[33]  Liang Dong,et al.  Emotion Recognition from Multiband EEG Signals Using CapsNet , 2019, Sensors.

[34]  Xiaoyan Zhu,et al.  Aspect-level Sentiment Analysis using AS-Capsules , 2019, WWW.

[35]  Naeem Ramzan,et al.  DREAMER: A Database for Emotion Recognition Through EEG and ECG Signals From Wireless Low-cost Off-the-Shelf Devices , 2018, IEEE Journal of Biomedical and Health Informatics.

[36]  Uma Shanker Tiwary,et al.  Affect representation and recognition in 3D continuous valence–arousal–dominance space , 2016, Multimedia Tools and Applications.

[37]  Bao-Liang Lu,et al.  Differential entropy feature for EEG-based vigilance estimation , 2013, 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[38]  Xiaodong Liu,et al.  Multi-Task Deep Neural Networks for Natural Language Understanding , 2019, ACL.

[39]  Shin-Dug Kim,et al.  Electroencephalography Based Fusion Two-Dimensional (2D)-Convolution Neural Networks (CNN) Model for Emotion Recognition System , 2018, Sensors.

[40]  Yong Peng,et al.  EEG-based emotion classification using deep belief networks , 2014, 2014 IEEE International Conference on Multimedia and Expo (ICME).

[41]  Engin Erzin,et al.  Monitoring Infant's Emotional Cry in Domestic Environments Using the Capsule Network Architecture , 2018, INTERSPEECH.

[42]  R. Dolan,et al.  Emotion, Cognition, and Behavior , 2002, Science.

[43]  Reda A. El-Khoribi,et al.  Emotion Recognition based on EEG using LSTM Recurrent Neural Network , 2017 .

[44]  P. Ekman.  An argument for basic emotions , 1992 .

[45]  Pradip Sircar,et al.  Automated emotion recognition based on higher order statistics and deep learning algorithm , 2020, Biomed. Signal Process. Control..

[46]  Egon L. van den Broek,et al.  Ubiquitous emotion-aware computing , 2011, Personal and Ubiquitous Computing.

[47]  Konstantinos N. Plataniotis,et al.  Brain Tumor Type Classification via Capsule Networks , 2018, 2018 25th IEEE International Conference on Image Processing (ICIP).

[48]  Dong Yu,et al.  Speech emotion recognition using deep neural network and extreme learning machine , 2014, INTERSPEECH.

[49]  B. Hjorth.  EEG analysis based on time domain properties. , 1970, Electroencephalography and clinical neurophysiology.

[50]  Xun Chen,et al.  Emotion Recognition From Multi-Channel EEG via Deep Forest , 2020, IEEE Journal of Biomedical and Health Informatics.

[51]  Guillermo Recio,et al.  Recognizing dynamic facial expressions of emotion: Specificity and intensity effects in event-related brain potentials , 2014, Biological Psychology.

[52]  Bao-Liang Lu,et al.  Investigating Critical Frequency Bands and Channels for EEG-Based Emotion Recognition with Deep Neural Networks , 2015, IEEE Transactions on Autonomous Mental Development.

[53]  Geoffrey E. Hinton,et al.  Transforming Auto-Encoders , 2011, ICANN.

[54]  Jason Weston,et al.  A unified architecture for natural language processing: deep neural networks with multitask learning , 2008, ICML '08.

[55]  James C. Christensen,et al.  Electrode replacement does not affect classification accuracy in dual-session use of a passive brain-computer interface for assessing cognitive workload , 2015, Front. Neurosci..

[56]  Mohsen Naji,et al.  Emotion classification during music listening from forehead biosignals , 2015, Signal Image Video Process..

[57]  Wenming Zheng,et al.  EEG Emotion Recognition Using Dynamical Graph Convolutional Neural Networks , 2020, IEEE Transactions on Affective Computing.