Multi-Domain Feature Fusion for Emotion Classification Using DEAP Dataset

Emotion recognition in real-time using electroencephalography (EEG) signals play a key role in human-computer interaction and affective computing. The existing emotion recognition models, that use stimuli such as music and pictures in controlled lab settings and limited number of emotion classes, have low ecological validity. Moreover, for effective emotion recognition identifying significant EEG features and electrodes is important. In our proposed model, we use the DEAP dataset consisting of physiological signals collected from 32 participants as they watched 40 movie (each of 60 seconds) clips. The main objective of this study is to explore multi-domain (time, wavelet, and frequency) features and hence, identify the set of stable features which contribute towards emotion classification catering to a larger number of emotion classes. Our proposed model is able to identify nine classes of emotions including happy, pleased, relaxed, excited, neutral, calm, distressed, miserable, and depressed with an average accuracy of 65.92%. Towards this end, we use support vector machine as a classifier along with 10-fold and leave-one-out cross-validation techniques. We achieve a significant emotion classification accuracy which could be vital towards developing solutions for affective computing and deal with a larger number of emotional states.

[1]  S. Kameyama,et al.  Asymmetric gelastic seizure as a lateralizing sign in patients with hypothalamic hamartoma , 2019, Epilepsy & Behavior.

[2]  Syed Muhammad Anwar,et al.  Emotion recognition in response to traditional and tactile enhanced multimedia using electroencephalography , 2018, Multimedia Tools and Applications.

[3]  Syed Muhammad Anwar,et al.  Emotion recognition from facial expressions using hybrid feature descriptors , 2018, IET Image Process..

[4]  Kin-yin. Mak,et al.  Neural bases of emotion regulation , 2009 .

[5]  Hung T. Nguyen,et al.  EEG-based emotion classification using innovative features and combined SVM and HMM classifier , 2017, 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[6]  Jennifer Healey,et al.  Toward Machine Emotional Intelligence: Analysis of Affective Physiological State , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[7]  Aydin Akan,et al.  Emotion recognition based on time-frequency distribution of EEG signals using multivariate synchrosqueezing transform , 2018, Digit. Signal Process..

[8]  Syed Muhammad Anwar,et al.  Human emotion recognition and analysis in response to audio music using brain signals , 2016, Comput. Hum. Behav..

[9]  Fumitoshi Matsuno,et al.  Feature Extraction from Subband Brain Signals and Its Classification , 2011 .

[10]  Bao-Liang Lu,et al.  Identifying Stable Patterns over Time for Emotion Recognition from EEG , 2016, IEEE Transactions on Affective Computing.

[11]  N. Kalin,et al.  Emotion, plasticity, context, and regulation: perspectives from affective neuroscience. , 2000, Psychological bulletin.

[12]  Wioleta Szwoch,et al.  Emotion Recognition Using Physiological Signals , 2015, MIDI '15.

[13]  Mahesh Panchal,et al.  A Review on Support Vector Machine for Data Classification , 2012 .

[14]  A. Nijholt,et al.  A survey of affective brain computer interfaces: principles, state-of-the-art, and challenges , 2014 .

[15]  Berat A. Erol,et al.  Toward Artificial Emotional Intelligence for Cooperative Social Human–Machine Interaction , 2020, IEEE Transactions on Computational Social Systems.

[16]  Syed Muhammad Anwar,et al.  Generation of personalized video summaries by detecting viewer's emotion using electroencephalography , 2019, J. Vis. Commun. Image Represent..

[17]  Mehmet Siraç Özerdem,et al.  Emotion recognition based on EEG features in movie clips with channel selection , 2017, Brain Informatics.

[18]  Abdulmotaleb El Saddik,et al.  Toward User-Independent Emotion Recognition Using Physiological Signals , 2019, IEEE Sensors Journal.

[19]  James Mountstephens,et al.  Deep learning for EEG-Based preference classification , 2017 .

[20]  R. E. Wheeler,et al.  Psychometric properties of resting anterior EEG asymmetry: temporal stability and internal consistency. , 1992, Psychophysiology.

[21]  Gernot R. Müller-Putz,et al.  Domain Adaptation Techniques for EEG-Based Emotion Recognition: A Comparative Study on Two Public Datasets , 2019, IEEE Transactions on Cognitive and Developmental Systems.

[22]  Seong Youb Chung,et al.  EEG-based emotion estimation using Bayesian weighted-log-posterior function and perceptron convergence algorithm , 2013, Comput. Biol. Medicine.

[23]  Soraia M. Alarcão,et al.  Emotions Recognition Using EEG Signals: A Survey , 2019, IEEE Transactions on Affective Computing.

[24]  A Gevins,et al.  Test–retest reliability of cognitive EEG , 2000, Clinical Neurophysiology.

[25]  Vinod Chandran,et al.  Evolutionary computation algorithms for feature selection of EEG-based emotion recognition using mobile sensors , 2018, Expert Syst. Appl..

[26]  Yong-Jin Liu,et al.  Real-Time Movie-Induced Discrete Emotion Recognition from EEG Signals , 2018, IEEE Transactions on Affective Computing.

[27]  Thierry Pun,et al.  DEAP: A Database for Emotion Analysis ;Using Physiological Signals , 2012, IEEE Transactions on Affective Computing.

[28]  Terrence J. Sejnowski,et al.  Utilizing Deep Learning Towards Multi-Modal Bio-Sensing and Vision-Based Affective Computing , 2019, IEEE Transactions on Affective Computing.

[29]  Bao-Liang Lu,et al.  Emotional state classification from EEG data using machine learning approach , 2014, Neurocomputing.

[30]  Klaus Wehrle,et al.  EEG-based automatic emotion recognition: Feature extraction, selection and classification methods , 2016, 2016 IEEE 18th International Conference on e-Health Networking, Applications and Services (Healthcom).

[31]  Rohit Prasad,et al.  Robust EEG emotion classification using segment level decision fusion , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[32]  Egon L. van den Broek,et al.  Ubiquitous emotion-aware computing , 2011, Personal and Ubiquitous Computing.

[33]  Yaacob Sazali,et al.  Classification of human emotion from EEG using discrete wavelet transform , 2010 .

[34]  David B. Kaber,et al.  Emotional State Classification in Patient–Robot Interaction Using Wavelet Analysis and Statistics-Based Feature Selection , 2013, IEEE Transactions on Human-Machine Systems.

[35]  Leo Galway,et al.  Towards emotion recognition for virtual environments: an evaluation of eeg features on benchmark dataset , 2017, Personal and Ubiquitous Computing.

[36]  M. Murugappan,et al.  Time-Frequency Analysis of EEG Signals for Human Emotion Detection , 2008 .

[37]  Samit Bhattacharya,et al.  Using Deep and Convolutional Neural Networks for Accurate Emotion Classification on DEAP Dataset , 2017, AAAI.

[38]  Syed Muhammad Anwar,et al.  Emotion Classification in Response to Tactile Enhanced Multimedia using Frequency Domain Features of Brain Signals , 2019, 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[39]  Luca Romeo,et al.  Multiple Instance Learning for Emotion Recognition Using Physiological Signals , 2022, IEEE Transactions on Affective Computing.

[40]  Shu-Ching Chen,et al.  Automatic Video Event Detection for Imbalance Data Using Enhanced Ensemble Deep Learning , 2017, Int. J. Semantic Comput..

[41]  Jonathan Klein,et al.  Computers that recognise and respond to user emotion: theoretical and practical implications , 2002, Interact. Comput..

[42]  Hui Song,et al.  Channel Selection Method for EEG Emotion Recognition Using Normalized Mutual Information , 2019, IEEE Access.

[43]  John Atkinson,et al.  Improving BCI-based emotion recognition by combining EEG feature selection and kernel classifiers , 2016, Expert Syst. Appl..

[44]  Syed Muhammad Anwar,et al.  A Hybrid Scheme for Drowsiness Detection Using Wearable Sensors , 2019, IEEE Sensors Journal.