A GAN-Based Data Augmentation Method for Multimodal Emotion Recognition

The lack of training data is an obstacle to build satisfactory multimodal emotion recognition models. Generative adversarial network (GAN) has recently shown great successes in generating realistic-like data. In this paper, we propose a GAN-based data augmentation method for enhancing the performance of multimodal emotion recognition models. We adopt conditional Boundary Equilibrium GAN (cBEGAN) to generate artificial differential entropy features of electroencephalography signal, eye movement data and their direct concatenations. The main advantage of cBEGAN is that it can overcome the instability of conventional GAN and has very quick converge speed. We evaluate our proposed method on two multimodal emotion datasets. The experimental results demonstrate that our proposed method achieves 4.6% and 8.9% improvements of mean accuracies on classifying three and five emotions, respectively.

[1]  Tonio Ball,et al.  EEG-GAN: Generative adversarial networks for electroencephalograhic (EEG) brain signals , 2018, ArXiv.

[2]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[3]  Fabien Lotte,et al.  Signal Processing Approaches to Minimize or Suppress Calibration Time in Oscillatory Activity-Based Brain–Computer Interfaces , 2015, Proceedings of the IEEE.

[4]  Xiao-Li Meng,et al.  The Art of Data Augmentation , 2001 .

[5]  Wei Liu,et al.  Emotion Recognition Using Multimodal Deep Learning , 2016, ICONIP.

[6]  Rui Li,et al.  Classification of Five Emotions from EEG and Eye Movement Signals: Complementary Representation Properties , 2019, 2019 9th International IEEE/EMBS Conference on Neural Engineering (NER).

[7]  Yun Luo,et al.  EEG Data Augmentation for Emotion Recognition Using a Conditional Wasserstein GAN , 2018, 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[8]  Bao-Liang Lu,et al.  Differential entropy feature for EEG-based emotion classification , 2013, 2013 6th International IEEE/EMBS Conference on Neural Engineering (NER).

[9]  Yan Liu,et al.  Data Augmentation for EEG-Based Emotion Recognition with Deep Convolutional Neural Networks , 2018, MMM.

[10]  Simon Osindero,et al.  Conditional Generative Adversarial Nets , 2014, ArXiv.

[11]  E. Vesterinen,et al.  Affective Computing , 2009, Encyclopedia of Biometrics.

[12]  Soumith Chintala,et al.  Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks , 2015, ICLR.

[13]  David Berthelot,et al.  BEGAN: Boundary Equilibrium Generative Adversarial Networks , 2017, ArXiv.

[14]  Bao-Liang Lu,et al.  Investigating Critical Frequency Bands and Channels for EEG-Based Emotion Recognition with Deep Neural Networks , 2015, IEEE Transactions on Autonomous Mental Development.

[15]  Yifei Lu,et al.  Combining Eye Movements and EEG to Enhance Emotion Recognition , 2015, IJCAI.

[16]  Mario Michael Krell,et al.  Rotational data augmentation for electroencephalographic data , 2017, 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).