Using Deep Convolutional Neural Network for Emotion Detection on a Physiological Signals Dataset (AMIGOS)

Recommender systems have been based on context and content, and now the technological challenge of making personalized recommendations based on the user emotional state arises through physiological signals that are obtained from devices or sensors. This paper applies the deep learning approach using a deep convolutional neural network on a dataset of physiological signals (electrocardiogram and galvanic skin response), in this case, the AMIGOS dataset. The detection of emotions is done by correlating these physiological signals with the data of arousal and valence of this dataset, to classify the affective state of a person. In addition, an application for emotion recognition based on classic machine learning algorithms is proposed to extract the features of physiological signals in the domain of time, frequency, and non-linear. This application uses a convolutional neural network for the automatic feature extraction of the physiological signals, and through fully connected network layers, the emotion prediction is made. The experimental results on the AMIGOS dataset show that the method proposed in this paper achieves a better precision of the classification of the emotional states, in comparison with the originally obtained by the authors of this dataset.

[1]  Andrew Ortony,et al.  The Cognitive Structure of Emotions , 1988 .

[2]  Samit Bhattacharya,et al.  Using Deep and Convolutional Neural Networks for Accurate Emotion Classification on DEAP Dataset , 2017, AAAI.

[3]  Daniel R. Fesenmaier,et al.  Measuring Human Senses and the Touristic Experience: Methods and Applications , 2017 .

[4]  Xiang Li,et al.  Emotion recognition from multi-channel EEG data through Convolutional Recurrent Neural Network , 2016, 2016 IEEE International Conference on Bioinformatics and Biomedicine (BIBM).

[5]  Byoung-Jun Park,et al.  Analysis of physiological signals for recognition of boredom, pain, and surprise emotions , 2015, Journal of Physiological Anthropology.

[6]  Rafael A. Calvo,et al.  Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications , 2010, IEEE Transactions on Affective Computing.

[7]  Giulio Gabrieli,et al.  PySiology: A Python Package for Physiological Feature Extraction , 2020, Neural Approaches to Dynamics of Signal Exchanges.

[8]  J. Russell MEASURES OF EMOTION , 1989 .

[9]  J. Gross,et al.  Emotion elicitation using films , 1995 .

[10]  Kimiaki Shirahama,et al.  Emotion Recognition Based on Physiological Sensor Data Using Codebook Approach , 2016, ITIB.

[11]  F. Shaffer,et al.  An Overview of Heart Rate Variability Metrics and Norms , 2017, Front. Public Health.

[12]  Matjaz Gams,et al.  Deep affect recognition from R-R intervals , 2017, UbiComp/ISWC Adjunct.

[13]  Koichi Kise,et al.  Emotional arousal estimation while reading comics based on physiological signal analysis , 2016, MANPU@ICPR.

[14]  Qiang Zhang,et al.  Classification of ECG signals based on 1D convolution neural network , 2017, 2017 IEEE 19th International Conference on e-Health Networking, Applications and Services (Healthcom).

[15]  R. Thayer The biopsychology of mood and arousal , 1989 .

[16]  Gyanendra K. Verma,et al.  Multimodal affect recognition in V-A-D space using deep learning , 2017, 2017 International Conference On Smart Technologies For Smart Nation (SmartTechCon).

[17]  J. Russell,et al.  A 12-Point Circumplex Structure of Core Affect. , 2011, Emotion.

[18]  Wei Liu,et al.  Emotion Recognition Using Multimodal Deep Learning , 2016, ICONIP.

[19]  Zhong Yin,et al.  Recognition of emotions using multimodal physiological signals and an ensemble deep learning model , 2017, Comput. Methods Programs Biomed..

[20]  Stéphane Bonnet,et al.  Selection of the Most Relevant Physiological Features for Classifying Emotion , 2015, PhyCS.

[21]  Yoshua Bengio,et al.  Learning deep physiological models of affect , 2013, IEEE Computational Intelligence Magazine.

[22]  Naeem Ramzan,et al.  DREAMER: A Database for Emotion Recognition Through EEG and ECG Signals From Wireless Low-cost Off-the-Shelf Devices , 2018, IEEE Journal of Biomedical and Health Informatics.

[23]  Enzo Pasquale Scilingo,et al.  Advances in Electrodermal Activity Processing with Applications for Mental Health , 2016, Springer International Publishing.

[24]  L. F. Barrett Discrete Emotions or Dimensions? The Role of Valence Focus and Arousal Focus , 1998 .

[25]  Subramanian Ramanathan,et al.  DECAF: MEG-Based Multimodal Database for Decoding Affective Physiological Responses , 2015, IEEE Transactions on Affective Computing.

[26]  Tapio Seppänen,et al.  Comparing features from ECG pattern and HRV analysis for emotion recognition system , 2016, 2016 IEEE Conference on Computational Intelligence in Bioinformatics and Computational Biology (CIBCB).

[27]  Stefan Winkler,et al.  ASCERTAIN: Emotion and Personality Recognition Using Commercial Sensors , 2018, IEEE Transactions on Affective Computing.

[28]  Thierry Pun,et al.  DEAP: A Database for Emotion Analysis ;Using Physiological Signals , 2012, IEEE Transactions on Affective Computing.

[29]  Murat Akçakaya,et al.  Decoding emotional experiences through physiological signal processing , 2016, 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[30]  Ioannis Patras,et al.  AMIGOS: A Dataset for Affect, Personality and Mood Research on Individuals and Groups , 2017, IEEE Transactions on Affective Computing.

[31]  B Pyakillya,et al.  Deep Learning for ECG Classification , 2017 .

[32]  Mohammad Soleymani,et al.  A Multimodal Database for Affect Recognition and Implicit Tagging , 2012, IEEE Transactions on Affective Computing.

[33]  Chengyu Liu,et al.  Evaluation of consistency of HRV indices change among different emotions , 2017, 2017 Chinese Automation Congress (CAC).

[34]  Akane Sano,et al.  Automatic identification of artifacts in electrodermal activity data , 2015, 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[35]  Karl J. Friston,et al.  Time-series analysis for rapid event-related skin conductance responses , 2009, Journal of Neuroscience Methods.

[36]  Zied Lachiri,et al.  Emotion Classification in Arousal Valence Model using MAHNOB-HCI Database , 2017 .

[37]  Dimitrios Buhalis,et al.  Smart Tourism Destinations Enhancing Tourism Experience Through Personalisation of Services , 2015, ENTER.

[38]  Zdzislaw Kowalczuk,et al.  Computational Approaches to Modeling Artificial Emotion – An Overview of the Proposed Solutions , 2016, Front. Robot. AI.

[39]  Rolf-Detlef Treede,et al.  Emotion Elicitation: A Comparison of Pictures and Films , 2016, Front. Psychol..

[40]  M. Bradley,et al.  The International Affective Picture System (IAPS) in the study of emotion and attention. , 2007 .

[41]  Chao Li,et al.  Deep Convolutional Neural Network for Emotion Recognition Using EEG and Peripheral Physiological Signal , 2017, ICIG.

[42]  Shima Okada,et al.  Fundamentals of Wearable Sensors for the Monitoring of Physical and Physiological Changes in Daily Life , 2014 .