Multi-method Fusion of Cross-Subject Emotion Recognition Based on High-Dimensional EEG Features

Emotion recognition using electroencephalogram (EEG) signals has attracted significant research attention. However, it is difficult to improve the emotional recognition effect across subjects. In response to this difficulty, in this study, multiple features were extracted for the formation of high-dimensional features. Based on the high-dimensional features, an effective method for cross-subject emotion recognition was then developed, which integrated the significance test/sequential backward selection and the support vector machine (ST-SBSSVM). The effectiveness of the ST-SBSSVM was validated on a dataset for emotion analysis using physiological signals (DEAP) and the SJTU Emotion EEG Dataset (SEED). With respect to high-dimensional features, the ST-SBSSVM average improved the accuracy of cross-subject emotion recognition by 12.4% on the DEAP and 26.5% on the SEED when compared with common emotion recognition methods. The recognition accuracy obtained using ST-SBSSVM was as high as that obtained using sequential backward selection (SBS) on the DEAP dataset. However, on the SEED dataset, the recognition accuracy increased by ~6% using ST-SBSSVM from that using the SBS. Using the ST-SBSSVM, ~97% (DEAP) and 91% (SEED) of the program runtime was eliminated when compared with the SBS. Compared with recent similar works, the method developed in this study for emotion recognition across all subjects was found to be effective, and its accuracy was 72% (DEAP) and 89% (SEED).

[1]  G. Schwartz,et al.  Differential lateralization for positive and negative emotion in the human brain: EEG spectral analysis , 1985, Neuropsychologia.

[2]  Aydin Akan,et al.  Emotion recognition from EEG signals by using multivariate empirical mode decomposition , 2018, Pattern Analysis and Applications.

[3]  Chi Zhang,et al.  Emotion Recognition from EEG Signals Using Multidimensional Information in EMD Domain , 2017, BioMed research international.

[4]  Yuhanis Yusof,et al.  OPTIMIZATION OF LEAST SQUARES SUPPORT VECTOR MACHINE TECHNIQUE USING GENETIC ALGORITHM FOR ELECTROENCEPHALOGRAM MULTI-DIMENSIONAL SIGNALS , 2016 .

[5]  Ram Bilas Pachori,et al.  Cross-Subject Emotion Recognition Using Flexible Analytic Wavelet Transform From EEG Signals , 2019, IEEE Sensors Journal.

[6]  Andrzej Cichocki,et al.  EmotionMeter: A Multimodal Framework for Recognizing Human Emotions , 2019, IEEE Transactions on Cybernetics.

[7]  Guillaume Chanel,et al.  Emotion Assessment From Physiological Signals for Adaptation of Game Difficulty , 2011, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[8]  J. Russell.  A circumplex model of affect. , 1980 .

[9]  M. Grigutsch,et al.  Music and emotion: electrophysiological correlates of the processing of pleasant and unpleasant music. , 2007, Psychophysiology.

[10]  Kazuhiko Takahashi.  Remarks on Emotion Recognition from Bio-Potential Signals , 2004 .

[11]  Bao-Liang Lu,et al.  A multimodal approach to estimating vigilance using EEG and forehead EOG , 2016, Journal of neural engineering.

[12]  Elisabeth André,et al.  Emotion recognition based on physiological changes in music listening , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  Léon J. M. Rothkrantz,et al.  Emotion recognition using brain activity , 2008, CompSysTech.

[14]  S. Hsieh,et al.  Classifying Different Emotional States by Means of EEG-Based Functional Connectivity Patterns , 2014, PloS one.

[15]  Houtan Jebelli,et al.  Mobile EEG-Based Workers’ Stress Recognition by Applying Deep Neural Network , 2018, Advances in Informatics and Computing in Civil and Construction Engineering.

[16]  L. Williams,et al.  Investigating models of affect: relationships among EEG alpha asymmetry, depression, and anxiety. , 2008, Emotion.

[17]  Stefan Carmien,et al.  Affective brain-computer interfaces: Psychophysiological markers of emotion in healthy persons and in persons with amyotrophic lateral sclerosis , 2009, 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops.

[18]  Jonghwa Kim,et al.  Bimodal Emotion Recognition using Speech and Physiological Changes , 2007 .

[19]  Ilias Maglogiannis,et al.  Intelligent Pervasive Healthcare Systems , 2008 .

[20]  B. Hjorth.  EEG analysis based on time domain properties. , 1970, Electroencephalography and clinical neurophysiology.

[21]  Bao-Liang Lu,et al.  EEG-based emotion recognition during watching movies , 2011, 2011 5th International IEEE/EMBS Conference on Neural Engineering.

[22]  Ram Bilas Pachori,et al.  Human Emotion Classification from EEG Signals Using Multiwavelet Transform , 2014, 2014 International Conference on Medical Biometrics.

[23]  Houtan Jebelli,et al.  Feasibility of Field Measurement of Construction Workers' Valence Using a Wearable EEG Device , 2017 .

[24]  Houtan Jebelli,et al.  EEG Signal-Processing Framework to Obtain High-Quality Brain Waves from an Off-the-Shelf Wearable EEG Device , 2018, J. Comput. Civ. Eng..

[25]  R. Plutchik.  The emotions: Facts, theories and a new model. , 1964 .

[26]  L. Pessoa,et al.  Emotion processing and the amygdala: from a 'low road' to 'many roads' of evaluating biological significance , 2010, Nature Reviews Neuroscience.

[27]  Dong Keun Kim,et al.  Arousal and Valence Classification Model Based on Long Short-Term Memory and DEAP Data for Mental Healthcare Management , 2018, Healthcare informatics research.

[28]  Bin Hu,et al.  Exploring EEG Features in Cross-Subject Emotion Recognition , 2018, Front. Neurosci..

[29]  Alejandra Figliola,et al.  Time-frequency analysis of electroencephalogram series. III. Wavelet packets and information cost function , 1998 .

[30]  Bao-Liang Lu,et al.  Cross-subject and Cross-gender Emotion Classification from EEG , 2015 .

[31]  Wei Zhang,et al.  Cross-Subject EEG Feature Selection for Emotion Recognition Using Transfer Recursive Feature Elimination , 2017, Front. Neurorobot..

[32]  Huiguang He,et al.  Multisource Transfer Learning for Cross-Subject EEG Emotion Recognition , 2020, IEEE Transactions on Cybernetics.

[33]  Jiayu Chen,et al.  Developing Construction Workers’ Mental Vigilance Indicators through Wavelet Packet Decomposition on EEG Signals , 2018 .

[34]  G. Knyazev,et al.  Gender differences in implicit and explicit processing of emotional facial expressions as revealed by event-related theta synchronization. , 2010, Emotion.

[35]  Thierry Pun,et al.  DEAP: A Database for Emotion Analysis ;Using Physiological Signals , 2012, IEEE Transactions on Affective Computing.

[36]  Jennifer Healey,et al.  Toward Machine Emotional Intelligence: Analysis of Affective Physiological State , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[37]  Olga Sourina,et al.  Real-Time Subject-Dependent EEG-Based Emotion Recognition Algorithm , 2014, Trans. Comput. Sci..

[38]  Olga Sourina,et al.  CogniMeter: EEG-based Emotion, Mental Workload and Stress Visual Monitoring , 2015, 2015 International Conference on Cyberworlds (CW).

[39]  Alan Hanjalic,et al.  Affective video content representation and modeling , 2005, IEEE Transactions on Multimedia.

[40]  SangHyun Lee,et al.  A Continuously Updated, Computationally Efficient Stress Recognition Framework Using Electroencephalogram (EEG) by Applying Online Multitask Learning Algorithms (OMTL) , 2019, IEEE Journal of Biomedical and Health Informatics.

[41]  J. Richman,et al.  Physiological time-series analysis using approximate entropy and sample entropy. , 2000, American journal of physiology. Heart and circulatory physiology.

[42]  Sheryl Brahnam and Lakhmi C. Jain.  Advanced computational intelligence paradigms in healthcare , 2013 .

[43]  Bao-Liang Lu,et al.  EEG-based vigilance estimation using extreme learning machines , 2013, Neurocomputing.

[44]  Jiayu Chen,et al.  Assessing Working Vulnerability of Construction Labor through EEG signal processing , 2016 .

[45]  Houtan Jebelli,et al.  Measuring Workers’ Emotional State during Construction Tasks Using Wearable EEG , 2018, Journal of Construction Engineering and Management.

[46]  Giulio Arcangeli,et al.  Work-related stress assessment in a population of Italian workers. The Stress Questionnaire. , 2015, The Science of the total environment.

[47]  Leontios J. Hadjileontiadis,et al.  A Novel Emotion Elicitation Index Using Frontal Brain Asymmetry for Enhanced EEG-Based Emotion Recognition , 2011, IEEE Transactions on Information Technology in Biomedicine.

[48]  Zhihong Zeng,et al.  A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[49]  M Murugappan,et al.  Physiological signals based human emotion Recognition: a review , 2011, 2011 IEEE 7th International Colloquium on Signal Processing and its Applications.

[50]  Houtan Jebelli,et al.  EEG-based workers' stress recognition at construction sites , 2018, Automation in Construction.

[51]  Petre Stoica,et al.  Spectral Analysis of Signals , 2009 .

[52]  Blake E. Ashforth,et al.  Emotion in the Workplace: A Reappraisal , 1995 .

[53]  Luc Van Gool,et al.  A 3-D Audio-Visual Corpus of Affective Communication , 2010, IEEE Transactions on Multimedia.

[54]  P Stömmer.  [Experimental studies of adipose tissue necrosis in acute tryptic pancreatitis]. , 1983, Leber, Magen, Darm.

[55]  Houtan Jebelli,et al.  A supervised learning-based construction workers' stress recognition using a wearable electroencephalography (EEG) device , 2018 .

[56]  Hung T. Nguyen,et al.  Investigation of window size in classification of EEG-emotion signal with wavelet entropy and support vector machine , 2015, 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[57]  Bao-Liang Lu,et al.  Investigating Critical Frequency Bands and Channels for EEG-Based Emotion Recognition with Deep Neural Networks , 2015, IEEE Transactions on Autonomous Mental Development.

[58]  T. Dalgleish,et al.  Handbook of cognition and emotion , 1999 .

[59]  Chao Li,et al.  Realization of stress detection using psychophysiological signals for improvement of human-computer interactions , 2005, Proceedings. IEEE SoutheastCon, 2005..