Spatio-Temporal Representation of an Electoencephalogram for Emotion Recognition Using a Three-Dimensional Convolutional Neural Network

Emotion recognition plays an important role in the field of human–computer interaction (HCI). An electroencephalogram (EEG) is widely used to estimate human emotion owing to its convenience and mobility. Deep neural network (DNN) approaches using an EEG for emotion recognition have recently shown remarkable improvement in terms of their recognition accuracy. However, most studies in this field still require a separate process for extracting handcrafted features despite the ability of a DNN to extract meaningful features by itself. In this paper, we propose a novel method for recognizing an emotion based on the use of three-dimensional convolutional neural networks (3D CNNs), with an efficient representation of the spatio-temporal representations of EEG signals. First, we spatially reconstruct raw EEG signals represented as stacks of one-dimensional (1D) time series data to two-dimensional (2D) EEG frames according to the original electrode position. We then represent a 3D EEG stream by concatenating the 2D EEG frames to the time axis. These 3D reconstructions of the raw EEG signals can be efficiently combined with 3D CNNs, which have shown a remarkable feature representation from spatio-temporal data. Herein, we demonstrate the accuracy of the emotional classification of the proposed method through extensive experiments on the DEAP (a Dataset for Emotion Analysis using EEG, Physiological, and video signals) dataset. Experimental results show that the proposed method achieves a classification accuracy of 99.11%, 99.74%, and 99.73% in the binary classification of valence and arousal, and, in four-class classification, respectively. We investigate the spatio-temporal effectiveness of the proposed method by comparing it to several types of input methods with 2D/3D CNN. We then verify the best performing shape of both the kernel and input data experimentally. We verify that an efficient representation of an EEG and a network that fully takes advantage of the data characteristics can outperform methods that apply handcrafted features.

[1]  Fei-Fei Li,et al.  Large-Scale Video Classification with Convolutional Neural Networks , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[2]  Bruno Verschuere,et al.  Psychopathy and Physiological Detection of Concealed Information: A review , 2006 .

[3]  Kyungha Min,et al.  A Multi-Column CNN Model for Emotion Recognition from EEG Signals , 2019, Sensors.

[4]  Seong Youb Chung,et al.  EEG-based emotion estimation using Bayesian weighted-log-posterior function and perceptron convergence algorithm , 2013, Comput. Biol. Medicine.

[5]  Victor Chang,et al.  Enhancing Text Using Emotion Detected from EEG Signals , 2018, Journal of Grid Computing.

[6]  Leo Galway,et al.  Towards emotion recognition for virtual environments: an evaluation of eeg features on benchmark dataset , 2017, Personal and Ubiquitous Computing.

[7]  Bao-Liang Lu,et al.  Investigating Critical Frequency Bands and Channels for EEG-Based Emotion Recognition with Deep Neural Networks , 2015, IEEE Transactions on Autonomous Mental Development.

[8]  Minsik Lee,et al.  Building a Compact Convolutional Neural Network for Embedded Intelligent Sensor Systems Using Group Sparsity and Knowledge Distillation , 2019, Sensors.

[9]  Martin Buss,et al.  Feature Extraction and Selection for Emotion Recognition from EEG , 2014, IEEE Transactions on Affective Computing.

[10]  Lan Huang,et al.  A Feature Extraction Method Based on Differential Entropy and Linear Discriminant Analysis for Emotion Recognition , 2019, Sensors.

[11]  Liang Dong,et al.  Emotion Recognition from Multiband EEG Signals Using CapsNet , 2019, Sensors.

[12]  Bao-Liang Lu,et al.  Differential entropy feature for EEG-based emotion classification , 2013, 2013 6th International IEEE/EMBS Conference on Neural Engineering (NER).

[13]  Yu Wang,et al.  EEG-Based Emotion Recognition with Deep Convolution Neural Network , 2019, 2019 IEEE 8th Data Driven Control and Learning Systems Conference (DDCLS).

[14]  Wei Gao,et al.  Pervasive and unobtrusive emotion sensing for human mental health , 2013, 2013 7th International Conference on Pervasive Computing Technologies for Healthcare and Workshops.

[15]  Clemens Brunner,et al.  A comparison of univariate, vector, bilinear autoregressive, and band power features for brain–computer interfaces , 2011, Medical & Biological Engineering & Computing.

[16]  S. Hyakin,et al.  Neural Networks: A Comprehensive Foundation , 1994 .

[17]  Zhenqi Li,et al.  SAE+LSTM: A New Framework for Emotion Recognition From Multi-Channel EEG , 2019, Front. Neurorobot..

[18]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[19]  Xiang Li,et al.  Emotion recognition from multi-channel EEG data through Convolutional Recurrent Neural Network , 2016, 2016 IEEE International Conference on Bioinformatics and Biomedicine (BIBM).

[20]  Lorenzo Torresani,et al.  Learning Spatiotemporal Features with 3D Convolutional Networks , 2014, 2015 IEEE International Conference on Computer Vision (ICCV).

[21]  Thierry Pun,et al.  DEAP: A Database for Emotion Analysis ;Using Physiological Signals , 2012, IEEE Transactions on Affective Computing.

[22]  Xiaowei Chen,et al.  Continuous Convolutional Neural Network with 3D Input for EEG-Based Emotion Recognition , 2018, ICONIP.

[23]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Huimin Lu,et al.  Facial Emotion Recognition Based on Biorthogonal Wavelet Entropy, Fuzzy Support Vector Machine, and Stratified Cross Validation , 2016, IEEE Access.

[25]  Gang Liu,et al.  An Ensemble Framework of Voice-Based Emotion Recognition System for Films and TV Programs , 2018, 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[26]  Samit Bhattacharya,et al.  Using Deep and Convolutional Neural Networks for Accurate Emotion Classification on DEAP Dataset , 2017, AAAI.

[27]  Ming Qiu,et al.  Emotion Recognition from Multi-Channel EEG through Parallel Convolutional Recurrent Neural Network , 2018, 2018 International Joint Conference on Neural Networks (IJCNN).

[28]  Reda A. El-Khoribi,et al.  Emotion Recognition based on EEG using LSTM Recurrent Neural Network , 2017 .

[29]  Yi Cao,et al.  EEG-Based Emotion Classification Using Spiking Neural Networks , 2020, IEEE Access.

[30]  Y. F. Huang,et al.  Accurate EEG-Based Emotion Recognition on Combined Features Using Deep Convolutional Neural Networks , 2019, IEEE Access.

[31]  Yann LeCun,et al.  A Closer Look at Spatiotemporal Convolutions for Action Recognition , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[32]  J. Russell,et al.  The circumplex model of affect: An integrative approach to affective neuroscience, cognitive development, and psychopathology , 2005, Development and Psychopathology.

[33]  Reda A. El-Khoribi,et al.  EEG-Based Emotion Recognition using 3D Convolutional Neural Networks , 2018 .

[34]  P. Lang.  The emotion probe. Studies of motivation and attention. , 1995, The American psychologist.

[35]  Roberto Sacile,et al.  Enhancing safety of transport by road by on-line monitoring of driver emotions , 2016, 2016 11th System of Systems Engineering Conference (SoSE).

[36]  Ling Huang,et al.  Feature Extraction of EEG Signals Using Power Spectral Entropy , 2008, 2008 International Conference on BioMedical Engineering and Informatics.

[37]  Erhan Ekmekcioglu,et al.  Investigating the Use of Pretrained Convolutional Neural Network on Cross-Subject and Cross-Dataset EEG Emotion Recognition , 2020, Sensors.

[38]  Tiago H. Falk,et al.  Deep learning-based electroencephalography analysis: a systematic review , 2019, Journal of neural engineering.

[39]  Youjun Li,et al.  Human Emotion Recognition with Electroencephalographic Multidimensional Features by Hybrid Deep Neural Networks , 2017 .

[40]  Bao-Liang Lu,et al.  Emotional state classification from EEG data using machine learning approach , 2014, Neurocomputing.

[41]  Yan Liu,et al.  Data Augmentation for EEG-Based Emotion Recognition with Deep Convolutional Neural Networks , 2018, MMM.

[42]  Sergey Ioffe,et al.  Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning , 2016, AAAI.

[43]  Zhenqi Li,et al.  Emotion Recognition from EEG Using RASM and LSTM , 2017, ICIMCS.

[44]  Chao Li,et al.  Deep Convolutional Neural Network for Emotion Recognition Using EEG and Peripheral Physiological Signal , 2017, ICIG.

[45]  Ram Bilas Pachori,et al.  Human Emotion Classification from EEG Signals Using Multiwavelet Transform , 2014, 2014 International Conference on Medical Biometrics.

[46]  Yutaka Satoh,et al.  Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet? , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[47]  Cordelia Schmid,et al.  Long-Term Temporal Convolutions for Action Recognition , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[48]  Tao Mei,et al.  Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[49]  Shin-Dug Kim,et al.  Electroencephalography Based Fusion Two-Dimensional (2D)-Convolution Neural Networks (CNN) Model for Emotion Recognition System , 2018, Sensors.