Parallel Sequence-Channel Projection Convolutional Neural Network for EEG-Based Emotion Recognition

One of the challenges in emotion recognition is finding an effective way to represent spatial-temporal features from EEG. To fully utilize the features on multiple dimensions of EEG signals, we propose a parallel sequence-channel projection convolutional neural network, including temporal stream sub-network, spatial stream sub-network, and fusion classification block. Temporal stream extracts temporal continuity via sequence-projection layer while spatial stream captures spatial correlation via channel-projection layer. Both sequence-projection and channel-projection adopt length-synchronized convolutional kernel to decode whole time and space information. The size of length-synchronized convolutional kernel is equal to the length of transmitted EEG sequence. The fusion classification block combines the extracted temporal and spatial features into a joint spatial-temporal feature vector for emotion prediction. In addition, we present a baseline noise filtering module to amplify input signals and a random channels exchange strategy to enrich the baseline-removed emotional signals. Experimental evaluation on DEAP dataset reveals that the proposed method achieves state-of-the-art classification performance for the binary classification task. The recognition accuracies reach to 96.16% and 95.89% for valence and arousal. The proposed method can improve 3% to 6% than other latest advanced works.

[1]  Brendan McCane,et al.  EmotioNet: A 3-D Convolutional Neural Network for EEG-based Emotion Recognition , 2018, 2018 International Joint Conference on Neural Networks (IJCNN).

[2]  Jun Guo,et al.  Feature selection for neutral vector in EEG signal classification , 2016, Neurocomputing.

[3]  Aydin Akan,et al.  Emotion recognition based on time-frequency distribution of EEG signals using multivariate synchrosqueezing transform , 2018, Digit. Signal Process..

[4]  George C. Runger,et al.  GCRNN: Group-Constrained Convolutional Recurrent Neural Network , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[5]  Soraia M. Alarcão,et al.  Emotions Recognition Using EEG Signals: A Survey , 2019, IEEE Transactions on Affective Computing.

[6]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[7]  Kaoru Hirota,et al.  Emotion recognition based on human gesture and speech information using RT middleware , 2011, 2011 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE 2011).

[8]  Peter E. Hart,et al.  Nearest neighbor pattern classification , 1967, IEEE Trans. Inf. Theory.

[9]  Yuan-Pin Lin,et al.  EEG-Based Emotion Recognition in Music Listening , 2010, IEEE Transactions on Biomedical Engineering.

[10]  H. Jasper,et al.  The ten-twenty electrode system of the International Federation. The International Federation of Clinical Neurophysiology. , 1999, Electroencephalography and clinical neurophysiology. Supplement.

[11]  C. L. Philip Chen,et al.  GCB-Net: Graph Convolutional Broad Network and Its Application in Emotion Recognition , 2019, IEEE Transactions on Affective Computing.

[12]  I. Guyon,et al.  Handwritten digit recognition: applications of neural network chips and automatic learning , 1989, IEEE Communications Magazine.

[13]  Theerawit Wilaiprasitporn,et al.  Consumer Grade EEG Measuring Sensors as Research Tools: A Review , 2020, IEEE Sensors Journal.

[14]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[15]  Peter W. McOwan,et al.  A real-time automated system for the recognition of human facial expressions , 2006, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[16]  Dylan Dah-Chuan Lu,et al.  School of Electrical and Information Engineering , 2013 .

[17]  Peng Chen,et al.  Performance Comparison of Machine Learning Algorithms for EEG-Signal-Based Emotion Recognition , 2017, ICANN.

[18]  S. Coleman,et al.  Emotion recognition with convolutional neural network and EEG-based EFDMs , 2020, Neuropsychologia.

[19]  Juan Cheng,et al.  EEG-Based Emotion Recognition via Channel-Wise Attention and Self Attention , 2023, IEEE Transactions on Affective Computing.

[20]  Anas Fattouh,et al.  Emotion estimation from EEG signals during listening to Quran using PSD features , 2016, 2016 7th International Conference on Computer Science and Information Technology (CSIT).

[21]  Theerawit Wilaiprasitporn,et al.  Consumer Grade Brain Sensing for Emotion Recognition , 2019, IEEE Sensors Journal.

[22]  Bao-Liang Lu,et al.  Investigating Critical Frequency Bands and Channels for EEG-Based Emotion Recognition with Deep Neural Networks , 2015, IEEE Transactions on Autonomous Mental Development.

[23]  Ginevra Castellano,et al.  Recognising Human Emotions from Body Movement and Gesture Dynamics , 2007, ACII.

[24]  Guanrong Chen,et al.  An adaptive optimal-Kernel time-frequency representation-based complex network method for characterizing fatigued behavior using the SSVEP-based BCI system , 2018, Knowl. Based Syst..

[25]  Minho Lee,et al.  ICA-Evolution Based Data Augmentation with Ensemble Deep Neural Networks Using Time and Frequency Kernels for Emotion Recognition from EEG-Data , 2019, IEEE Transactions on Affective Computing.

[26]  Thierry Pun,et al.  DEAP: A Database for Emotion Analysis ;Using Physiological Signals , 2012, IEEE Transactions on Affective Computing.

[27]  Zhaoxiang Zhang,et al.  Hierarchical Convolutional Neural Networks for EEG-Based Emotion Recognition , 2017, Cognitive Computation.

[28]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[29]  W. G. Parrott,et al.  Emotions in social psychology : essential readings , 2001 .

[30]  J. Russell.  A circumplex model of affect. , 1980 .

[31]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[32]  Qingming Huang,et al.  Image Saliency Detection Video Saliency Detection Co-saliency Detection Temporal RGBD Saliency Detection Motion , 2018 .

[33]  Qing Zhang,et al.  Autonomous emotion development using incremental modified adaptive neuro-fuzzy inference system , 2012, Neurocomputing.

[34]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[35]  P. Ekman,et al.  DIFFERENCES Universals and Cultural Differences in the Judgments of Facial Expressions of Emotion , 2004 .

[36]  Sam Kwong,et al.  Efficient In-Loop Filtering Based on Enhanced Deep Convolutional Neural Networks for HEVC , 2020, IEEE Transactions on Image Processing.

[37]  Shaoguo Wen,et al.  Fine-Grained Vehicle Classification With Channel Max Pooling Modified CNNs , 2019, IEEE Transactions on Vehicular Technology.

[38]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[39]  Hong Gu,et al.  Emotion Recognition Based on Double Tree Complex Wavelet Transform and Machine Learning in Internet of Things , 2019, IEEE Access.

[40]  Qingming Huang,et al.  Going From RGB to RGBD Saliency: A Depth-Guided Transformation Model , 2020, IEEE Transactions on Cybernetics.

[41]  Wenming Zheng,et al.  EEG Emotion Recognition Using Dynamical Graph Convolutional Neural Networks , 2020, IEEE Transactions on Affective Computing.

[42]  Douglas Kline,et al.  Revisiting squared-error and cross-entropy functions for training neural network classifiers , 2005, Neural Computing & Applications.

[43]  Hasan Al-Nashash,et al.  Emotion Recognition Based on Fusion of Local Cortical Activations and Dynamic Functional Networks Connectivity: An EEG Study , 2019, IEEE Access.

[44]  Bao-Liang Lu,et al.  Differential entropy feature for EEG-based emotion classification , 2013, 2013 6th International IEEE/EMBS Conference on Neural Engineering (NER).

[45]  Sungho Jo,et al.  Deep Physiological Affect Network for the Recognition of Human Emotions , 2020, IEEE Transactions on Affective Computing.

[46]  Dongmei Jiang,et al.  Emotion recognition from spatiotemporal EEG representations with hybrid convolutional recurrent neural networks via wearable multi-channel headset , 2020, Comput. Commun..

[47]  Tong Zhang,et al.  Spatial–Temporal Recurrent Neural Network for Emotion Recognition , 2017, IEEE Transactions on Cybernetics.

[48]  Seong Youb Chung,et al.  EEG-based emotion estimation using Bayesian weighted-log-posterior function and perceptron convergence algorithm , 2013, Comput. Biol. Medicine.

[49]  Martin Buss,et al.  Feature Extraction and Selection for Emotion Recognition from EEG , 2014, IEEE Transactions on Affective Computing.

[50]  Bao-Liang Lu,et al.  Identifying Stable Patterns over Time for Emotion Recognition from EEG , 2016, IEEE Transactions on Affective Computing.

[51]  Feng Duan,et al.  A Novel Deep Learning Approach With Data Augmentation to Classify Motor Imagery Signals , 2019, IEEE Access.

[52]  Javad Frounchi,et al.  Wavelet-based emotion recognition system using EEG signal , 2017, Neural Computing and Applications.

[53]  Fu Lee Wang,et al.  Frame-level Bit Allocation Optimization Based on Video Content Characteristics for HEVC , 2020, ACM Trans. Multim. Comput. Commun. Appl..

[54]  Theerawit Wilaiprasitporn,et al.  Affective EEG-Based Person Identification Using the Deep Learning Approach , 2018, IEEE Transactions on Cognitive and Developmental Systems.

[55]  Ronald J. Williams,et al.  A Learning Algorithm for Continually Running Fully Recurrent Neural Networks , 1989, Neural Computation.

[56]  Rafael A. Calvo,et al.  Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications , 2010, IEEE Transactions on Affective Computing.

[57]  C. L. Philip Chen,et al.  I-Ching Divination Evolutionary Algorithm and its Convergence Analysis , 2017, IEEE Transactions on Cybernetics.

[58]  Ming Qiu,et al.  Emotion Recognition from Multi-Channel EEG through Parallel Convolutional Recurrent Neural Network , 2018, 2018 International Joint Conference on Neural Networks (IJCNN).