Subject-Independent Emotion Recognition During Music Listening Based on EEG Using Deep Convolutional Neural Networks

Emotion recognition during music listening using electroencephalogram (EEG) has gained more attention from researchers, recently. Many studies focused on accuracy on one subject while subject-independent performance evaluation was still unclear. In this paper, the objective is to create an emotion recognition model that can be applied to multiple subjects. By adopting convolutional neural networks (CNNs), advantage could be gained from utilizing information from electrodes and time steps. Using CNNs also does not need feature extraction which might leave out other related but unobserved features. CNNs with three to seven convolutional layers were deployed in this research. We measured their performance with a binary classification task for compositions of emotions including arousal and valence. The results showed that our method captured EEG signal patterns from numerous subjects by 10-fold cross validation with 81.54% and 86.87% accuracy from arousal and valence respectively. The method also showed a higher capability of generalization to unseen subjects than the previous method as can be observed from the results of leave-one-subject-out validation.

[1]  Mohammad Soleymani,et al.  Analysis of EEG Signals and Facial Expressions for Continuous Emotion Detection , 2016, IEEE Transactions on Affective Computing.

[2]  Reda A. El-Khoribi,et al.  Emotion Recognition based on EEG using LSTM Recurrent Neural Network , 2017 .

[3]  Miyoung Kim,et al.  A Review on the Computational Methods for Emotional State Estimation from the Human EEG , 2013, Comput. Math. Methods Medicine.

[4]  Rosa Gil,et al.  Method for Improving EEG Based Emotion Recognition by Combining It with Synchronized Biometric and Eye Tracking Technologies in a Non-invasive and Low Cost Way , 2016, Front. Comput. Neurosci..

[5]  Thierry Pun,et al.  DEAP: A Database for Emotion Analysis ;Using Physiological Signals , 2012, IEEE Transactions on Affective Computing.

[6]  Gyanendra K. Verma,et al.  Multimodal fusion framework: A multiresolution approach for emotion classification and recognition from physiological signals , 2014, NeuroImage.

[7]  H. Kandemir,et al.  Effects of Physical Activity Based HCI Games on the Attention, Emotion and Sensory-Motor Coordination , 2018, Advances in Service and Industrial Robotics.

[8]  Masayuki Numao,et al.  Multimodal Fusion of EEG and Musical Features in Music-Emotion Recognition , 2017, AAAI.

[9]  Arnaud Delorme,et al.  EEGLAB, SIFT, NFT, BCILAB, and ERICA: New Tools for Advanced EEG Processing , 2011, Comput. Intell. Neurosci..

[10]  Xiang Li,et al.  Emotion recognition from multi-channel EEG data through Convolutional Recurrent Neural Network , 2016, 2016 IEEE International Conference on Bioinformatics and Biomedicine (BIBM).

[11]  Zahra Khalili,et al.  Emotion recognition system using brain and peripheral signals: Using correlation dimension to improve the results of EEG , 2009, 2009 International Joint Conference on Neural Networks.

[12]  Yuan-Pin Lin,et al.  EEG-Based Emotion Recognition in Music Listening , 2010, IEEE Transactions on Biomedical Engineering.

[13]  B. Matthews.  Comparison of the predicted and observed secondary structure of T4 phage lysozyme. , 1975, Biochimica et biophysica acta.

[14]  J. Russell.  A circumplex model of affect. , 1980 .

[15]  Stefan Koelsch,et al.  Brain and Music , 2012 .

[16]  Ruifeng Xu,et al.  A novel convolutional neural networks for emotion recognition based on EEG signal , 2017, 2017 International Conference on Security, Pattern Analysis, and Cybernetics (SPAC).

[17]  Mohammad Soleymani,et al.  A Multimodal Database for Affect Recognition and Implicit Tagging , 2012, IEEE Transactions on Affective Computing.

[18]  Pasin Israsena,et al.  EEG-Based Emotion Recognition Using Deep Learning Network with Principal Component Based Covariate Shift Adaptation , 2014, TheScientificWorldJournal.

[19]  T. Jung,et al.  Fusion of electroencephalographic dynamics and musical contents for estimating emotional responses in music listening , 2014, Front. Neurosci..

[20]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[21]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[22]  Shiuan Huang,et al.  Depression Detection Using Relative EEG Power Induced by Emotionally Positive Images and a Conformal Kernel Support Vector Machine , 2018, Applied Sciences.

[23]  Jia-Hao Wu,et al.  A personalized music recommendation system based on electroencephalography feedback , 2017, Multimedia Tools and Applications.

[24]  S. Koelsch.  Brain correlates of music-evoked emotions , 2014, Nature Reviews Neuroscience.

[25]  Shin Ishii,et al.  Multi-Sensor Based State Prediction for Personal Mobility Vehicles , 2016, PLoS ONE.

[26]  Rafael A. Calvo,et al.  Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications , 2010, IEEE Transactions on Affective Computing.