An overlapping sliding window and combined features based emotion recognition system for EEG signals

PurposeThe purpose of this study is to propose an alternative efficient 3D emotion recognition model for variable-length electroencephalogram (EEG) data.Design/methodology/approachClassical AMIGOS data set which comprises of multimodal records of varying lengths on mood, personality and other physiological aspects on emotional response is used for empirical assessment of the proposed overlapping sliding window (OSW) modelling framework. Two features are extracted using Fourier and Wavelet transforms: normalised band power (NBP) and normalised wavelet energy (NWE), respectively. The arousal, valence and dominance (AVD) emotions are predicted using one-dimension (1D) and two-dimensional (2D) convolution neural network (CNN) for both single and combined features.FindingsThe two-dimensional convolution neural network (2D CNN) outcomes on EEG signals of AMIGOS data set are observed to yield the highest accuracy, that is 96.63%, 95.87% and 96.30% for AVD, respectively, which is evidenced to be at least 6% higher as compared to the other available competitive approaches.Originality/valueThe present work is focussed on the less explored, complex AMIGOS (2018) data set which is imbalanced and of variable length. EEG emotion recognition-based work is widely available on simpler data sets. The following are the challenges of the AMIGOS data set addressed in the present work: handling of tensor form data; proposing an efficient method for generating sufficient equal-length samples corresponding to imbalanced and variable-length data.; selecting a suitable machine learning/deep learning model; improving the accuracy of the applied model.

[1]  Ioannis Patras,et al.  AMIGOS: A Dataset for Affect, Personality and Mood Research on Individuals and Groups , 2017, IEEE Transactions on Affective Computing.

[2]  Chunyan Miao,et al.  EEG-Based Emotion Recognition Using Regularized Graph Neural Networks , 2019, IEEE Transactions on Affective Computing.

[3]  Jeffrey R. Powell,et al.  Transgenic Aedes aegypti Mosquitoes Transfer Genes into a Natural Population , 2019, Scientific Reports.

[4]  Dan Liu,et al.  An effective feature extraction method by power spectral density of EEG signal for 2-class motor imagery-based BCI , 2018, Medical & Biological Engineering & Computing.

[5]  Junzhong Zou,et al.  Automatic epileptic EEG classification based on differential entropy and attention model , 2020, Eng. Appl. Artif. Intell..

[6]  Mustafa Ulutas,et al.  Classification of EEG signals using time domain features , 2015, 2015 23nd Signal Processing and Communications Applications Conference (SIU).

[7]  Uma Shanker Tiwary,et al.  Affect representation and recognition in 3D continuous valence–arousal–dominance space , 2016, Multimedia Tools and Applications.

[8]  An-Yeu Wu,et al.  Entropy-Assisted Multi-Modal Emotion Recognition Framework Based on Physiological Signals , 2018, 2018 IEEE-EMBS Conference on Biomedical Engineering and Sciences (IECBES).

[9]  Yonghong Tan,et al.  Strengthen EEG-based emotion recognition using firefly integrated optimization algorithm , 2020, Appl. Soft Comput..

[10]  Yuan-Pin Lin,et al.  EEG-Based Emotion Recognition in Music Listening , 2010, IEEE Transactions on Biomedical Engineering.

[11]  Ingrid Daubechies,et al.  Ten Lectures on Wavelets , 1992 .

[12]  Juan Cheng,et al.  EEG-Based Emotion Recognition via Channel-Wise Attention and Self Attention , 2023, IEEE Transactions on Affective Computing.

[13]  Zhen Cui,et al.  Variational Instance-Adaptive Graph for EEG Emotion Recognition , 2021, IEEE Transactions on Affective Computing.

[14]  Fakhri Karray,et al.  Survey on speech emotion recognition: Features, classification schemes, and databases , 2011, Pattern Recognit..

[15]  Yue Wang,et al.  Heart sound signals can be used for emotion recognition , 2019, Scientific Reports.

[16]  Mladen Russo,et al.  Emotion recognition based on EEG feature maps through deep learning network , 2021, Engineering Science and Technology, an International Journal.

[17]  Chao Li,et al.  Exploring temporal representations by leveraging attention-based bidirectional LSTM-RNNs for multi-modal emotion recognition , 2020, Inf. Process. Manag..

[18]  Enas Abdulhay,et al.  Using Deep Convolutional Neural Network for Emotion Detection on a Physiological Signals Dataset (AMIGOS) , 2019, IEEE Access.

[19]  Xiaoou Li,et al.  Support vector machine classification for large data sets via minimum enclosing ball clustering , 2008, Neurocomputing.

[20]  Leontios J. Hadjileontiadis,et al.  Emotion Recognition From EEG Using Higher Order Crossings , 2010, IEEE Transactions on Information Technology in Biomedicine.

[21]  Varun Bajaj,et al.  Time–Frequency Representation and Convolutional Neural Network-Based Emotion Recognition , 2020, IEEE Transactions on Neural Networks and Learning Systems.

[22]  Olga Sourina,et al.  Real-Time Fractal-Based Valence Level Recognition from EEG , 2013, Trans. Comput. Sci..

[23]  Kongqiao Wang,et al.  EEG-based emotion recognition using an end-to-end regional-asymmetric convolutional neural network , 2020, Knowl. Based Syst..

[24]  Tzyy-Ping Jung,et al.  Multi-modal Approach for Affective Computing , 2018, 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[25]  S. Coleman,et al.  Emotion recognition with convolutional neural network and EEG-based EFDMs , 2020, Neuropsychologia.

[26]  Rui Zhang,et al.  Predicting Inter-session Performance of SMR-Based Brain–Computer Interface Using the Spectral Entropy of Resting-State EEG , 2015, Brain Topography.

[27]  Guillermo Recio,et al.  Recognizing dynamic facial expressions of emotion: Specificity and intensity effects in event-related brain potentials , 2014, Biological Psychology.

[28]  Chin-Teng Lin,et al.  EEG-Based Brain-Computer Interfaces (BCIs): A Survey of Recent Studies on Signal Sensing Technologies and Computational Intelligence Approaches and Their Applications , 2020, IEEE/ACM Transactions on Computational Biology and Bioinformatics.

[29]  Xun Chen,et al.  Multi-channel EEG-based emotion recognition via a multi-level features guided capsule network , 2020, Comput. Biol. Medicine.

[30]  Quoc V. Le.  A Tutorial on Deep Learning Part 2: Autoencoders, Convolutional Neural Networks and Recurrent Neural Networks , 2015 .

[31]  Ashish Khanna,et al.  Emotion Recognition and Detection Methods: A Comprehensive Survey , 2020, Journal of Artificial Intelligence and Systems.

[32]  J. Russell.  A circumplex model of affect. , 1980 .

[33]  Pradip Sircar,et al.  Automated emotion recognition based on higher order statistics and deep learning algorithm , 2020, Biomed. Signal Process. Control..

[34]  Terrence J. Sejnowski,et al.  Utilizing Deep Learning Towards Multi-Modal Bio-Sensing and Vision-Based Affective Computing , 2019, IEEE Transactions on Affective Computing.

[35]  Hatice Gunes,et al.  Bi-modal emotion recognition from expressive face and body gestures , 2007, J. Netw. Comput. Appl..

[36]  Nitesh V. Chawla,et al.  SMOTE: Synthetic Minority Over-sampling Technique , 2002, J. Artif. Intell. Res..

[37]  Xinchun Cui,et al.  EEG emotion recognition using fusion model of graph convolutional neural networks and LSTM , 2021, Appl. Soft Comput..

[38]  Amjed S. Al-Fahoum,et al.  Methods of EEG Signal Features Extraction Using Linear Analysis in Frequency and Time-Frequency Domains , 2014, ISRN neuroscience.

[39]  Soraia M. Alarcão,et al.  Emotions Recognition Using EEG Signals: A Survey , 2019, IEEE Transactions on Affective Computing.