Emotional EEG classification using connectivity features and convolutional neural networks

Convolutional neural networks (CNNs) are widely used to recognize the user's state through electroencephalography (EEG) signals. In the previous studies, the EEG signals are usually fed into the CNNs in the form of high-dimensional raw data. However, this approach makes it difficult to exploit the brain connectivity information that can be effective in describing the functional brain network and estimating the perceptual state of the user. We introduce a new classification system that utilizes brain connectivity with a CNN and validate its effectiveness via the emotional video classification by using three different types of connectivity measures. Furthermore, two data-driven methods to construct the connectivity matrix are proposed to maximize classification performance. Further analysis reveals that the level of concentration of the brain connectivity related to the emotional property of the target video is correlated with classification performance.

[1]  N. Frijda The laws of emotion. , 1988, The American psychologist.

[2]  Ying Zeng,et al.  EEG Based Emotion Recognition by Combining Functional Connectivity Network and Local Activations , 2019, IEEE Transactions on Biomedical Engineering.

[3]  Derek Abbott,et al.  Feature Extraction and Selection , 2012 .

[4]  A. Friederici,et al.  Investigating emotion with music: An fMRI study , 2006, Human brain mapping.

[5]  Thierry Pun,et al.  DEAP: A Database for Emotion Analysis ;Using Physiological Signals , 2012, IEEE Transactions on Affective Computing.

[6]  Zhaoxiang Zhang,et al.  Hierarchical Convolutional Neural Networks for EEG-Based Emotion Recognition , 2017, Cognitive Computation.

[7]  Carlos M Travieso,et al.  EEG biometric identification: a thorough exploration of the time-frequency domain. , 2015, Journal of neural engineering.

[8]  R. Dolan,et al.  Emotion, Cognition, and Behavior , 2002, Science.

[9]  Angelo Gemignani,et al.  The dynamics of EEG gamma responses to unpleasant visual stimuli: From local activity to functional connectivity , 2012, NeuroImage.

[10]  Reda A. El-Khoribi,et al.  Emotion Recognition based on EEG using LSTM Recurrent Neural Network , 2017 .

[11]  Sahar Moghimi,et al.  Toward automatic detection of brain responses to emotional music through analysis of EEG effective connectivity , 2016, Comput. Hum. Behav..

[12]  Seong-Eun Moon,et al.  Convolutional Neural Network Approach for Eeg-Based Emotion Recognition Using Brain Connectivity and its Spatial Information , 2018, 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[13]  Elisabeth André,et al.  Emotion recognition based on physiological changes in music listening , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  F. Varela,et al.  Measuring phase synchrony in brain signals , 1999, Human brain mapping.

[15]  B. Hjorth EEG analysis based on time domain properties. , 1970, Electroencephalography and clinical neurophysiology.

[16]  Senem Velipasalar,et al.  Building predictive models of emotion with functional near-infrared spectroscopy , 2018, Int. J. Hum. Comput. Stud..

[17]  S. Hsieh,et al.  Classifying Different Emotional States by Means of EEG-Based Functional Connectivity Patterns , 2014, PloS one.

[18]  Bao-Liang Lu,et al.  Investigating Critical Frequency Bands and Channels for EEG-Based Emotion Recognition with Deep Neural Networks , 2015, IEEE Transactions on Autonomous Mental Development.

[19]  J. Leeuw Applications of Convex Analysis to Multidimensional Scaling , 2000 .

[20]  K. Berridge,et al.  Unconscious Emotion , 2004 .

[21]  Mahmoud Hassan,et al.  Electroencephalography Source Connectivity: Aiming for High Resolution of Brain Networks in Time and Space , 2018, IEEE Signal Processing Magazine.

[22]  Joseph T. Lizier,et al.  JIDT: An Information-Theoretic Toolkit for Studying the Dynamics of Complex Systems , 2014, Front. Robot. AI.

[23]  Yong-Jin Liu,et al.  Real-Time Movie-Induced Discrete Emotion Recognition from EEG Signals , 2018, IEEE Transactions on Affective Computing.

[24]  Seong-Eun Moon,et al.  Implicit Analysis of Perceptual Multimedia Experience Based on Physiological Response: A Review , 2017, IEEE Transactions on Multimedia.

[25]  Aydin Akan,et al.  Emotion recognition based on time-frequency distribution of EEG signals using multivariate synchrosqueezing transform , 2018, Digit. Signal Process..

[26]  Tiago H. Falk,et al.  Mutual information between inter-hemispheric EEG spectro-temporal patterns: A new feature for automated affect recognition , 2015, 2015 7th International IEEE/EMBS Conference on Neural Engineering (NER).

[27]  Suhua Zhang,et al.  An approach to EEG-based emotion recognition using combined feature extraction method , 2016, Neuroscience Letters.

[28]  Human emotion. , 1978, Nebraska Symposium on Motivation. Nebraska Symposium on Motivation.

[29]  Youjun Li,et al.  Human Emotion Recognition with Electroencephalographic Multidimensional Features by Hybrid Deep Neural Networks , 2017 .

[30]  Xin Zhao,et al.  Study on an effective cross-stimulus emotion recognition model using EEGs based on feature selection and support vector machine , 2016, International Journal of Machine Learning and Cybernetics.

[31]  John J. B. Allen,et al.  Frontal EEG asymmetry as a moderator and mediator of emotion , 2004, Biological Psychology.

[32]  P. Ekman,et al.  DIFFERENCES Universals and Cultural Differences in the Judgments of Facial Expressions of Emotion , 2004 .

[33]  Jane-Ling Wang,et al.  Stringing High-Dimensional Data for Functional Analysis , 2011 .

[34]  Christian Mühl,et al.  EEG analysis for implicit tagging of video data , 2009, 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops.

[35]  Schreiber,et al.  Measuring information transfer , 2000, Physical review letters.

[36]  Subramanian Ramanathan,et al.  DECAF: MEG-Based Multimodal Database for Decoding Affective Physiological Responses , 2015, IEEE Transactions on Affective Computing.

[37]  Junichiro Yoshimoto,et al.  Connectivity inference from neural recording data: Challenges, mathematical bases and research directions , 2017, Neural Networks.

[38]  Jonas K. Olofsson,et al.  Affective picture processing: An integrative review of ERP findings , 2008, Biological Psychology.

[39]  I. K. Wood,et al.  Neuroscience: Exploring the brain , 1996 .

[40]  M. Cabanac What is emotion? , 2002, Behavioural Processes.

[41]  Chun-Jui Chen,et al.  A New Approach for Functional Connectivity via Alignment of Blood Oxygen Level-Dependent Signals , 2019, Brain Connect..

[42]  Seong-Eun Moon,et al.  Eeg-Based Video Identification Using Graph Signal Modeling and Graph Convolutional Neural Network , 2018, 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[43]  M. Peters,et al.  Volume conduction effects in EEG and MEG. , 1998, Electroencephalography and clinical neurophysiology.

[44]  Martin Buss,et al.  Feature Extraction and Selection for Emotion Recognition from EEG , 2014, IEEE Transactions on Affective Computing.

[45]  Petia D. Koprinkova-Hristova,et al.  Learning to decode human emotions with Echo State Networks , 2016, Neural Networks.

[46]  Tommaso Costa,et al.  EEG phase synchronization during emotional response to positive and negative film stimuli , 2006, Neuroscience Letters.

[47]  John J. B. Allen,et al.  Frontal asymmetry as a mediator and moderator of emotion: An updated review. , 2018, Psychophysiology.

[48]  Filippo Zappasodi,et al.  Hemispheric asymmetries and emotions: Evidence from effective connectivity , 2018, Neuropsychologia.

[49]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.