Emotion Estimation from EEG - A Dual Deep Learning Approach Combined with Saliency

Emotion estimation is an active field of research that has an important impact on the interaction between human and computer. Among the different modality to assess emotion, electroencephalogram (EEG) representing the electrical brain activity presented motivating results during the last decade. Emotion estimation from EEG could help in the diagnosis or rehabilitation of certain diseases. In this paper, we propose a dual method considering the physiological knowledge defined by specialists combined with novel deep learning (DL) models initially dedicated to computer vision. The joint learning has been enhanced with model saliency analysis. To present a global approach, the model has been evaluated on four publicly available datasets and achieves similar results to the state-of-theart approaches and outperforming results for two of the proposed datasets with a lower standard deviation that reflects higher stability. For sake of reproducibility, the codes and models proposed in this paper are available at github.com/VDelv/Emotion-EEG.

[1]  Jose Javier Gonzalez Ortiz,et al.  What is the State of Neural Network Pruning? , 2020, MLSys.

[2]  Jeffrey Mark Siskind,et al.  The Perils and Pitfalls of Block Design for EEG Classification Experiments , 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[3]  Mohammed Yeasin,et al.  Learning Representations from EEG with Deep Recurrent-Convolutional Neural Networks , 2015, ICLR.

[4]  Chunyan Miao,et al.  EEG-Based Emotion Recognition Using Regularized Graph Neural Networks , 2019, IEEE Transactions on Affective Computing.

[5]  René Vidal,et al.  Latent Space Sparse Subspace Clustering , 2013, 2013 IEEE International Conference on Computer Vision.

[6]  Wenming Zheng,et al.  MPED: A Multi-Modal Physiological Emotion Database for Discrete Emotion Recognition , 2019, IEEE Access.

[7]  Lei Wang,et al.  A Novel Bi-Hemispheric Discrepancy Model for EEG Emotion Recognition , 2019, IEEE Transactions on Cognitive and Developmental Systems.

[8]  Christian Barillot,et al.  The first MICCAI challenge on PET tumor segmentation , 2018, Medical Image Anal..

[9]  Tong Zhang,et al.  A Novel Neural Network Model based on Cerebral Hemispheric Asymmetry for EEG Emotion Recognition , 2018, IJCAI.

[10]  Yong Du,et al.  Hierarchical recurrent neural network for skeleton based action recognition , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  P. Werbos,et al.  Beyond Regression : "New Tools for Prediction and Analysis in the Behavioral Sciences , 1974 .

[12]  Bao-Liang Lu,et al.  Investigating Critical Frequency Bands and Channels for EEG-Based Emotion Recognition with Deep Neural Networks , 2015, IEEE Transactions on Autonomous Mental Development.

[13]  Bao-Liang Lu,et al.  Differential entropy feature for EEG-based emotion classification , 2013, 2013 6th International IEEE/EMBS Conference on Neural Engineering (NER).

[14]  Bao-Liang Lu,et al.  Multimodal emotion recognition using EEG and eye tracking data , 2014, 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[15]  Ana B. Porto-Pazos,et al.  Deep Artificial Neural Networks and Neuromorphic Chips for Big Data Analysis: Pharmaceutical and Bioinformatics Applications , 2016, International journal of molecular sciences.

[16]  Geoffrey E. Hinton,et al.  Dynamic Routing Between Capsules , 2017, NIPS.

[17]  G. Schwartz,et al.  Differential lateralization for positive and negative emotion in the human brain: EEG spectral analysis , 1985, Neuropsychologia.

[18]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[19]  Dejing Dou,et al.  A deep learning approach for human behavior prediction with explanations in health social networks: social restricted Boltzmann machine (SRBM+) , 2016, Social Network Analysis and Mining.

[20]  Ah-Hwee Tan,et al.  EEG-based Emotion Recognition Using Spatial-Temporal Representation via Bi-GRU , 2020, 2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC).

[21]  S. Muthukumaraswamy High-frequency brain activity and muscle artifacts in MEG/EEG: a review and recommendations , 2013, Front. Hum. Neurosci..

[22]  W. Wasserman,et al.  Genome-wide prediction of cis-regulatory regions using supervised deep learning methods , 2016, BMC Bioinformatics.

[23]  Thierry Pun,et al.  DEAP: A Database for Emotion Analysis ;Using Physiological Signals , 2012, IEEE Transactions on Affective Computing.

[24]  Yongtian He,et al.  Deep learning for electroencephalogram (EEG) classification tasks: a review , 2019, Journal of neural engineering.

[25]  Andrew Zisserman,et al.  Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps , 2013, ICLR.

[26]  Andrzej Cichocki,et al.  EmotionMeter: A Multimodal Framework for Recognizing Human Emotions , 2019, IEEE Transactions on Cybernetics.

[27]  Wenming Zheng,et al.  EEG Emotion Recognition Using Dynamical Graph Convolutional Neural Networks , 2020, IEEE Transactions on Affective Computing.

[28]  Victor S. Lempitsky,et al.  Unsupervised Domain Adaptation by Backpropagation , 2014, ICML.