A novel convolution bi-directional gated recurrent unit neural network for emotion recognition in multichannel electroencephalogram signals.

BACKGROUND Recognising emotions in humans is a great challenge in the present era and has several applications under affective computing. Deep learning (DL) found a success tool for predict for emotions in different modalities. OBJECTIVE To predict 3D emotions with high accuracy in multichannel physiological signals, i.e. electroencephalogram (EEG). METHODS A hybrid DL model consist of CNN and GRU is proposed in this work for emotion recognition in EEG recordings. A convolution neural network (CNN) has the capability of learning abstract representation, whereas gated recurrent units (GRU) have the capability of exploring temporal correlation. A bi-directional variation of GRU is used here to learn features in both directions. Discrete and dimensional emotion indices are recognised in two publicly available datasets namely SEED and DREAMER, respectively. A fused feature of energy and Shannon entropy (𝐸𝑛𝑆𝐸→) and energy and differential entropy (𝐸𝑛𝐷𝐸→) features are fed to the proposed classifier to improve the efficiency of the model. RESULTS The performance of the presented model is measured in terms of average accuracy, which is obtained as 86.9% and 93.9% for SEED and DREAMER datasets, respectively. CONCLUSION The proposed convolution bi-directional gated recurrent unit neural network (CNN-BiGRU) model outperforms most of the state-of-the-art and competitive hybrid DL models, which indicates the effectiveness of emotion recognition using EEG signals and provides a scientific base for the implementation of human-computer interaction (HCI).

[1]  Tawfik Al Hadhrami,et al.  Deep Learning-Based Approach for Emotion Recognition Using Electroencephalography (EEG) Signals Using Bi-Directional Long Short-Term Memory (Bi-LSTM) , 2022, Sensors.

[2]  Fatma Patlar Akbulut Hybrid deep convolutional model-based emotion recognition using multiple physiological signals , 2022, Computer methods in biomechanics and biomedical engineering.

[3]  J. Sengupta,et al.  Ensemble Machine Learning-Based Affective Computing for Emotion Recognition Using Dual-Decomposed EEG Signals , 2022, IEEE Sensors Journal.

[4]  Ying Li,et al.  Learning correlation filter with fused feature and reliable response for real-time tracking , 2022, Journal of Real-Time Image Processing.

[5]  R. Jarina,et al.  End-to-end music emotion variation detection using iteratively reconstructed deep features , 2022, Multimedia Tools and Applications.

[6]  Abhijit Bhattacharyya,et al.  Human emotion recognition based on time-frequency analysis of multivariate EEG signal , 2021, Knowl. Based Syst..

[7]  Mahesh Raveendranatha Panicker,et al.  Subject Independent Emotion Recognition using EEG Signals Employing Attention Driven Neural Networks , 2021, Biomed. Signal Process. Control..

[8]  Xue Wang,et al.  Infrared and Visible Image Fusion via Decoupling Network , 2022, IEEE Transactions on Instrumentation and Measurement.

[9]  S. Ge,et al.  Command Filter-Based Adaptive Fuzzy Finite-Time Output Feedback Control of Nonlinear Electrohydraulic Servo System , 2022, IEEE Transactions on Instrumentation and Measurement.

[10]  Chuangquan Chen,et al.  Fusing Frequency-Domain Features and Brain Connectivity Features for Cross-Subject Emotion Recognition , 2022, IEEE Transactions on Instrumentation and Measurement.

[11]  S. R. Shahamiri,et al.  A new classification system for autism based on machine learning of artificial intelligence. , 2021, Technology and health care : official journal of the European Society for Engineering and Medicine.

[12]  Mingjiang Wang,et al.  EEG-Based Emotion Recognition by Exploiting Fused Network Entropy Measures of Complex Networks across Subjects , 2021, Entropy.

[13]  Abdulhamit Subasi,et al.  EEG-based emotion recognition using tunable Q wavelet transform and rotation forest ensemble classifier , 2021, Biomed. Signal Process. Control..

[14]  Mladen Russo,et al.  Emotion recognition based on EEG feature maps through deep learning network , 2021, Engineering Science and Technology, an International Journal.

[15]  Dongdong Li,et al.  EEG Emotion Recognition Based on 3-D Feature Representation and Dilated Fully Convolutional Networks , 2021, IEEE Transactions on Cognitive and Developmental Systems.

[16]  Shuaiqi Liu,et al.  Subject-Independent Emotion Recognition of EEG Signals Based on Dynamic Empirical Convolutional Neural Network , 2020, IEEE/ACM Transactions on Computational Biology and Bioinformatics.

[17]  Xun Chen,et al.  Emotion Recognition From Multi-Channel EEG via Deep Forest , 2020, IEEE Journal of Biomedical and Health Informatics.

[18]  Seong-Whan Lee,et al.  eRAD-Fe: Emotion Recognition-Assisted Deep Learning Framework , 2021, IEEE Transactions on Instrumentation and Measurement.

[19]  Vaishali M. Joshi,et al.  EEG based emotion detection using fourth order spectral moment and deep learning , 2021, Biomed. Signal Process. Control..

[20]  Md. Atiqur Rahman Ahad,et al.  Emotion Recognition From EEG Signal Focusing on Deep Learning and Shallow Learning Techniques , 2021, IEEE Access.

[21]  Junshan Li,et al.  Speech emotion recognition model based on Bi-GRU and Focal Loss , 2020, Pattern Recognit. Lett..

[22]  D. Şengür,et al.  Efficient approach for EEG‐based emotion recognition , 2020, Electronics Letters.

[23]  Nejra Beganovic,et al.  FPGA-based real-time epileptic seizure classification using Artificial Neural Network , 2020, Biomed. Signal Process. Control..

[24]  Yu Song,et al.  EEG based emotion recognition using fusion feature extraction method , 2020, Multimedia Tools and Applications.

[25]  Humaira Nisar,et al.  Comparison of different feature extraction methods for EEG-based emotion recognition , 2020 .

[26]  Jianhua Zhang,et al.  Emotion recognition using multi-modal data and machine learning techniques: A tutorial and review , 2020, Inf. Fusion.

[27]  Wenming Zheng,et al.  EEG Emotion Recognition Using Dynamical Graph Convolutional Neural Networks , 2020, IEEE Transactions on Affective Computing.

[28]  Leonardo Cunha de Miranda,et al.  Brain–Computer Interface Games Based on Consumer-Grade EEG Devices: A Systematic Literature Review , 2019, Int. J. Hum. Comput. Interact..

[29]  Válber César Cavalcanti Roza,et al.  Multimodal Approach for Emotion Recognition Based on Simulated Flight Experiments , 2019, Sensors.

[30]  Rhee,et al.  A High-Accuracy Model Average Ensemble of Convolutional Neural Networks for Classification of Cloud Image Patches on Small Datasets , 2019, Applied Sciences.

[31]  Yongqiang Cheng,et al.  Interpretable Emotion Recognition Using EEG Signals , 2019, IEEE Access.

[32]  Y. F. Huang,et al.  Accurate EEG-Based Emotion Recognition on Combined Features Using Deep Convolutional Neural Networks , 2019, IEEE Access.

[33]  Xu Zhang,et al.  Chinese medical question answer selection via hybrid models based on CNN and GRU , 2019, Multimedia Tools and Applications.

[34]  J. X. Chen,et al.  A Hierarchical Bidirectional GRU Model With Attention for EEG-Based Emotion Classification , 2019, IEEE Access.

[35]  Sten Hanke,et al.  Emotion Recognition from Physiological Signal Analysis: A Review , 2019, BRAINS/WS-AFFIN@AmI.

[36]  Theerawit Wilaiprasitporn,et al.  EDOSE: Emotion Datasets from Open Source EEG with a Real-Time Bracelet Sensor , 2018, ArXiv.

[37]  Alex Sherstinsky,et al.  Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network , 2018, Physica D: Nonlinear Phenomena.

[38]  E. Custovic,et al.  An Expert Diagnostic System to Automatically Identify Asthma and Chronic Obstructive Pulmonary Disease in Clinical Settings , 2018, Scientific Reports.

[39]  Ali Momennezhad EEG-based emotion recognition utilizing wavelet coefficients , 2018, Multimedia Tools and Applications.

[40]  Lejla Gurbeta,et al.  Application of Neural Networks for classification of Patau, Edwards, Down, Turner and Klinefelter Syndrome based on first trimester maternal serum screening data, ultrasonographic findings and patient demographics , 2018, BMC Medical Genomics.

[41]  H. Critchley,et al.  The influence of physiological signals on cognition , 2018, Current Opinion in Behavioral Sciences.

[42]  Dinesh Singh,et al.  EEG Based Emotion Classification Mechanism in BCI , 2018 .

[43]  Naeem Ramzan,et al.  DREAMER: A Database for Emotion Recognition Through EEG and ECG Signals From Wireless Low-cost Off-the-Shelf Devices , 2018, IEEE Journal of Biomedical and Health Informatics.

[44]  M. Z. Soroush,et al.  A Review on EEG Signals Based Emotion Recognition , 2017 .

[45]  M. Feidakis A Review of Emotion-Aware Systems for e-Learning in Virtual Environments , 2016 .

[46]  Javad Frounchi,et al.  Wavelet-based emotion recognition system using EEG signal , 2017, Neural Computing and Applications.

[47]  Bao-Liang Lu,et al.  Investigating Critical Frequency Bands and Channels for EEG-Based Emotion Recognition with Deep Neural Networks , 2015, IEEE Transactions on Autonomous Mental Development.

[48]  B. de Gelder,et al.  The perception of emotion in body expressions. , 2015, Wiley interdisciplinary reviews. Cognitive science.

[49]  Toyoaki Nishida,et al.  Using physiological signals to detect natural interactive behavior , 2010, Applied Intelligence.

[50]  Yuan-Pin Lin,et al.  EEG-Based Emotion Recognition in Music Listening , 2010, IEEE Transactions on Biomedical Engineering.

[51]  Rafael A. Calvo,et al.  Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications , 2010, IEEE Transactions on Affective Computing.

[52]  Jennifer Healey,et al.  Toward Machine Emotional Intelligence: Analysis of Affective Physiological State , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[53]  L. Feldman Valence Focus and Arousal Focus: Individual Differences in the Structure of Affective Experience , 1995 .

[54]  Ross A. Thompson,et al.  Emotion regulation: a theme in search of definition. , 1994, Monographs of the Society for Research in Child Development.

[55]  Lawrence D. Jackel,et al.  Backpropagation Applied to Handwritten Zip Code Recognition , 1989, Neural Computation.

[56]  Takayuki Ito,et al.  Neocognitron: A neural network model for a mechanism of visual pattern recognition , 1983, IEEE Transactions on Systems, Man, and Cybernetics.

[57]  J. Russell A circumplex model of affect. , 1980 .