A Bi-Hemisphere Domain Adversarial Neural Network Model for EEG Emotion Recognition

In this paper, we propose a novel neural network model, called bi-hemisphere domain adversarial neural network (BiDANN) model, for electroencephalograph (EEG) emotion recognition. The BiDANN model is inspired by the neuroscience findings that the left and right hemispheres of human's brain are asymmetric to the emotional response. It contains a global and two local domain discriminators that work adversarially with a classifier to learn discriminative emotional features for each hemisphere. At the same time, it tries to reduce the possible domain differences in each hemisphere between source and target domains so as to improve the generality of the recognition model. In addition, we also propose an improved version of BiDANN, denoted by BiDANN-S, for subject-independent EEG emotion recognition problem by lowering the influences of the personal information of subjects to the EEG emotion recognition. Extensive experiments on the SEED database are conducted to evaluate the performance of both BiDANN and BiDANN-S. The experimental results have shown that the proposed BiDANN and BiDANN models achieve state-of-the-art performance in the EEG emotion recognition.

[1]  Jennifer C. Britton,et al.  Neural correlates of social and nonsocial emotions: An fMRI study , 2006, NeuroImage.

[2]  Fernando De la Torre,et al.  Selective Transfer Machine for Personalized Facial Expression Analysis , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[3]  L. Brusco,et al.  Hemispheric specialization in affective responses, cerebral dominance for language, and handedness Lateralization of emotion, language, and dexterity , 2015, Behavioural Brain Research.

[4]  Johan A. K. Suykens,et al.  Least Squares Support Vector Machine Classifiers , 1999, Neural Processing Letters.

[5]  A. Nijholt,et al.  A survey of affective brain computer interfaces: principles, state-of-the-art, and challenges , 2014 .

[6]  Hiie Hinrikus,et al.  Electroencephalographic spectral asymmetry index for detection of depression , 2009, Medical & Biological Engineering & Computing.

[7]  Xavier Bresson,et al.  Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering , 2016, NIPS.

[8]  R. Davidson.  Cerebral asymmetry, emotion, and affective style. , 1995 .

[9]  R. Dolan,et al.  Emotion, Cognition, and Behavior , 2002, Science.

[10]  Ivor W. Tsang,et al.  Domain Adaptation via Transfer Component Analysis , 2009, IEEE Transactions on Neural Networks.

[11]  Bao-Liang Lu,et al.  Personalizing EEG-Based Affective Models with Transfer Learning , 2016, IJCAI.

[12]  Yuan Shi,et al.  Geodesic flow kernel for unsupervised domain adaptation , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[13]  François Laviolette,et al.  Domain-Adversarial Training of Neural Networks , 2015, J. Mach. Learn. Res..

[14]  Takafumi Kanamori,et al.  A Least-squares Approach to Direct Importance Estimation , 2009, J. Mach. Learn. Res..

[15]  P. Ekman,et al.  Approach-withdrawal and cerebral asymmetry: emotional expression and brain physiology. I. , 1990, Journal of personality and social psychology.

[16]  Nicu Sebe,et al.  Recurrent Face Aging , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  N. Geschwind,et al.  Cerebral lateralization. Biological mechanisms, associations, and pathology: I. A hypothesis and a program for research. , 1985, Archives of neurology.

[18]  Michel J A M van Putten,et al.  The revised brain symmetry index. , 2007, Clinical neurophysiology : official journal of the International Federation of Clinical Neurophysiology.

[19]  S. Dimond,et al.  Differing emotional response from right and left hemispheres , 1976, Nature.

[20]  R. Davidson,et al.  Depression: perspectives from affective neuroscience. , 2002, Annual review of psychology.

[21]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[22]  Motoaki Kawanabe,et al.  Direct Importance Estimation with Model Selection and Its Application to Covariate Shift Adaptation , 2007, NIPS.

[23]  Jason Weston,et al.  Large Scale Transductive SVMs , 2006, J. Mach. Learn. Res..

[24]  Hema Swetha Koppula,et al.  Recurrent Neural Networks for driver activity anticipation via sensory-fusion architecture , 2015, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[25]  M. Putten,et al.  Behavioral measures and EEG monitoring using the Brain Symmetry Index during the Wada test in children , 2012, Epilepsy & Behavior.

[26]  Wenming Zheng,et al.  EEG Emotion Recognition Using Dynamical Graph Convolutional Neural Networks , 2020, IEEE Transactions on Affective Computing.

[27]  Nicu Sebe,et al.  We are not All Equal: Personalizing Models for Facial Expression Analysis with Transductive Parameter Transfer , 2014, ACM Multimedia.

[28]  Zhen Cui,et al.  EEG Emotion Recognition Based on Graph Regularized Sparse Linear Regression , 2018, Neural Processing Letters.

[29]  Bao-Liang Lu,et al.  Investigating Critical Frequency Bands and Channels for EEG-Based Emotion Recognition with Deep Neural Networks , 2015, IEEE Transactions on Autonomous Mental Development.

[30]  Bernhard Schölkopf,et al.  Nonlinear Component Analysis as a Kernel Eigenvalue Problem , 1998, Neural Computation.

[31]  Tinne Tuytelaars,et al.  Unsupervised Visual Domain Adaptation Using Subspace Alignment , 2013, 2013 IEEE International Conference on Computer Vision.

[32]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[33]  Philip S. Yu,et al.  Domain Invariant Transfer Kernel Learning , 2015, IEEE Transactions on Knowledge and Data Engineering.

[34]  Miyoung Kim,et al.  A Review on the Computational Methods for Emotional State Estimation from the Human EEG , 2013, Comput. Math. Methods Medicine.

[35]  Qiuping Xu.  Canonical correlation Analysis , 2014 .

[36]  Mert R. Sabuncu,et al.  A Surface-based Analysis of Language Lateralization and Cortical Asymmetry , 2013, Journal of Cognitive Neuroscience.

[37]  Ehsan Lotfi,et al.  Practical emotional neural networks , 2014, Neural Networks.

[38]  Alan C. Evans,et al.  Functional localization and lateralization of human olfactory cortex , 1992, Nature.

[39]  Wenming Zheng,et al.  Multichannel EEG-Based Emotion Recognition via Group Sparse Canonical Correlation Analysis , 2017, IEEE Transactions on Cognitive and Developmental Systems.

[40]  Bin Hu,et al.  Exploring EEG Features in Cross-Subject Emotion Recognition , 2018, Front. Neurosci..

[41]  Léon Bottou,et al.  Large-Scale Machine Learning with Stochastic Gradient Descent , 2010, COMPSTAT.

[42]  K. R. Seeja,et al.  Emotional State Recognition with EEG Signals Using Subject Independent Approach , 2018, WIR.

[43]  Toshimitsu Musha,et al.  Feature extraction from EEGs associated with emotions , 1997, Artificial Life and Robotics.

[44]  T. Egner,et al.  Emotional processing in anterior cingulate and medial prefrontal cortex , 2011, Trends in Cognitive Sciences.

[45]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[46]  D. Tucker.  Lateral brain function, emotion, and conceptualization. , 1981, Psychological bulletin.

[47]  Laurent Albera,et al.  Emotion Recognition Based on High-Resolution EEG Recordings and Reconstructed Brain Sources , 2020, IEEE Transactions on Affective Computing.

[48]  Martin Buss,et al.  Feature Extraction and Selection for Emotion Recognition from EEG , 2014, IEEE Transactions on Affective Computing.

[49]  Kristen A. Lindquist,et al.  A functional architecture of the human brain: emerging insights from the science of emotion , 2012, Trends in Cognitive Sciences.

[50]  Tong Zhang,et al.  A Novel Neural Network Model based on Cerebral Hemispheric Asymmetry for EEG Emotion Recognition , 2018, IJCAI.

[51]  Trevor Darrell,et al.  DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition , 2013, ICML.