The Design of CNN Architectures for Optimal Six Basic Emotion Classification Using Multiple Physiological Signals

This study aimed to design an optimal emotion recognition method using multiple physiological signal parameters acquired by bio-signal sensors for improving the accuracy of classifying individual emotional responses. Multiple physiological signals such as respiration (RSP) and heart rate variability (HRV) were acquired in an experiment from 53 participants when six basic emotion states were induced. Two RSP parameters were acquired from a chest-band respiration sensor, and five HRV parameters were acquired from a finger-clip blood volume pulse (BVP) sensor. A newly designed deep-learning model based on a convolutional neural network (CNN) was adopted for detecting the identification accuracy of individual emotions. Additionally, the signal combination of the acquired parameters was proposed to obtain high classification accuracy. Furthermore, a dominant factor influencing the accuracy was found by comparing the relativeness of the parameters, providing a basis for supporting the results of emotion classification. The users of this proposed model will soon be able to improve the emotion recognition model further based on CNN using multimodal physiological signals and their sensors.

[1]  Simon J. Julier,et al.  DeepBreath: Deep learning of breathing patterns for automatic stress recognition using low-cost thermal imaging in unconstrained settings , 2017, 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII).

[2]  Theekshana Dissanayake,et al.  An Ensemble Learning Approach for Electrocardiogram Sensor Based Human Emotion Recognition , 2019, Sensors.

[3]  Sylvia D. Kreibig,et al.  Autonomic nervous system activity in emotion: A review , 2010, Biological Psychology.

[4]  Jean-Marc Vesin,et al.  Using cardio-respiratory signals to recognize emotions elicited by watching music video clips , 2016, 2016 IEEE 18th International Workshop on Multimedia Signal Processing (MMSP).

[5]  Xiaodan Zhuang,et al.  Compact unsupervised EEG response representation for emotion recognition , 2014, IEEE-EMBS International Conference on Biomedical and Health Informatics (BHI).

[6]  Fernando De la Rosa,et al.  Identification of multimodal signals for emotion recognition in the context of human-robot interaction , 2018 .

[7]  Xianxiang Chen,et al.  Respiration-based emotion recognition with deep learning , 2017, Comput. Ind..

[8]  Tal Hassner,et al.  Emotion Recognition in the Wild via Convolutional Neural Networks and Mapped Binary Patterns , 2015, ICMI.

[9]  Mauricio A. Álvarez,et al.  Gaussian process dynamical models for multimodal affect recognition , 2016, 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[10]  Mohammad I. Daoud,et al.  EEG-Based Emotion Recognition Using Quadratic Time-Frequency Distribution , 2018, Sensors.

[11]  Luca Citi,et al.  Revealing Real-Time Emotional Responses: a Personalized Assessment based on Heartbeat Dynamics , 2014, Scientific Reports.

[12]  Mohamed Chetouani,et al.  Hilbert-Huang transform based physiological signals analysis for emotion recognition , 2009, 2009 IEEE International Symposium on Signal Processing and Information Technology (ISSPIT).

[13]  Dimitrios Hatzinakos,et al.  ECG Pattern Analysis for Emotion Detection , 2012, IEEE Transactions on Affective Computing.

[14]  Kyoung Shin Park,et al.  Mediating individual affective experience through the emotional photo frame , 2015, J. Ambient Intell. Smart Environ..

[15]  Weishan Zhang,et al.  Emotion Recognition from Chinese Speech for Smart Affective Services Using a Combination of SVM and DBN , 2017, Sensors.

[16]  Zhenqi Li,et al.  A Review of Emotion Recognition Using Physiological Signals , 2018, Sensors.

[17]  Jonghwa Kim,et al.  PPG Filtering Method for Respiration Measurement in U-Health Care System , 2008 .

[18]  Rafael A. Calvo,et al.  Effect of Experimental Factors on the Recognition of Affective Mental States through Physiological Measures , 2009, Australasian Conference on Artificial Intelligence.

[19]  John Tudor,et al.  Fuzzy logic based emotion classification , 2014, 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[20]  U. Rajendra Acharya,et al.  Heart rate variability: a review , 2006, Medical and Biological Engineering and Computing.

[21]  Christine L. Lisetti,et al.  Emotion recognition from physiological signals using wireless sensors for presence technologies , 2004, Cognition, Technology & Work.

[22]  Yongjoo Cho,et al.  Interactive emotional content communications system using portable wireless biofeedback device , 2011, IEEE Transactions on Consumer Electronics.

[23]  F. Shaffer,et al.  An Overview of Heart Rate Variability Metrics and Norms , 2017, Front. Public Health.

[24]  Dong Keun Kim,et al.  Real-Time Mobile Emotional Content Player Using Smartphone Camera-Based PPG Measurement , 2019, Mob. Inf. Syst..

[25]  Martin Stevens.  Multimodal Signals and Communication , 2013 .

[26]  Jianhua Zhang,et al.  Physiological Feature Based Emotion Recognition via an Ensemble Deep Autoencoder with Parsimonious Structure , 2017 .

[27]  Yanning Zhang,et al.  Hybrid Deep Neural Network--Hidden Markov Model (DNN-HMM) Based Speech Emotion Recognition , 2013, 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction.

[28]  F. Shaffer,et al.  A healthy heart is not a metronome: an integrative review of the heart's anatomy and heart rate variability , 2014, Front. Psychol..

[29]  Gyanendra K. Verma,et al.  Multimodal fusion framework: A multiresolution approach for emotion classification and recognition from physiological signals , 2014, NeuroImage.

[30]  Zhong Yin,et al.  Recognition of emotions using multimodal physiological signals and an ensemble deep learning model , 2017, Comput. Methods Programs Biomed..

[31]  P. Ekman,et al.  Voluntary facial action generates emotion-specific autonomic nervous system activity. , 1990, Psychophysiology.

[32]  Yong Peng,et al.  EEG-based emotion classification using deep belief networks , 2014, 2014 IEEE International Conference on Multimedia and Expo (ICME).

[33]  Leontios J. Hadjileontiadis,et al.  Emotion Recognition from Brain Signals Using Hybrid Adaptive Filtering and Higher Order Crossings Analysis , 2010, IEEE Transactions on Affective Computing.

[34]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[35]  Maozhen Li,et al.  Emotion detection from EEG recordings , 2016, 2016 12th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery (ICNC-FSKD).

[36]  Bao-Liang Lu,et al.  Identifying Stable Patterns over Time for Emotion Recognition from EEG , 2016, IEEE Transactions on Affective Computing.

[37]  Samit Bhattacharya,et al.  Using Deep and Convolutional Neural Networks for Accurate Emotion Classification on DEAP Dataset , 2017, AAAI.

[38]  Min Cheol Whang,et al.  Interactive emotional lighting system using physiological signals , 2013, IEEE Transactions on Consumer Electronics.

[39]  Kiavash Bahreini,et al.  A fuzzy logic approach to reliable real-time recognition of facial emotions , 2019, Multimedia Tools and Applications.

[40]  Wei Liu,et al.  Emotion Recognition Using Multimodal Deep Learning , 2016, ICONIP.

[41]  Wanhui Wen,et al.  Emotion Recognition Based on Multi-Variant Correlation of Physiological Signals , 2014, IEEE Transactions on Affective Computing.

[42]  K. Kim.  EMOTIONAL LIGHTING SYSTEM ABLE TO EMOTION REASONING USING FUZZY INFERENCE , 2018 .

[43]  Z. Piotrowski,et al.  Prediction of fatigue and sleep onset using HRV analysis , 2012, Proceedings of the 19th International Conference Mixed Design of Integrated Circuits and Systems - MIXDES 2012.

[44]  Johannes Wagner,et al.  From Physiological Signals to Emotions: Implementing and Comparing Selected Methods for Feature Extraction and Classification , 2005, 2005 IEEE International Conference on Multimedia and Expo.

[45]  J. Russell.  A circumplex model of affect. , 1980 .

[46]  Wei Zhang,et al.  Emotion recognition by assisted learning with convolutional neural networks , 2018, Neurocomputing.

[47]  Sun K. Yoo,et al.  A Deep Neural Network-Based Pain Classifier Using a Photoplethysmography Signal , 2019, Sensors.

[48]  Matti Pietikäinen,et al.  Multi-modal emotion analysis from facial expressions and electroencephalogram , 2016, Comput. Vis. Image Underst..

[49]  P. Ekman.  The argument and evidence about universals in facial expressions of emotion. , 1989 .

[50]  Toshihiro Hiraoka,et al.  Heart Rate Variability-Based Driver Drowsiness Detection and Its Validation With EEG , 2019, IEEE Transactions on Biomedical Engineering.

[51]  J. Sohn.  Review on Discrete, Appraisal, and Dimensional Models of Emotion , 2011 .

[52]  Yu Cao,et al.  PNN for EEG-based Emotion Recognition , 2016, 2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC).

[53]  Cristian A. Torres-Valencia,et al.  Comparative analysis of physiological signals and electroencephalogram (EEG) for multimodal emotion recognition using generative models , 2014, 2014 XIX Symposium on Image, Signal Processing and Artificial Vision.

[54]  Heather T. Ma,et al.  Emotion recognition based on the multiple physiological signals , 2016, 2016 IEEE International Conference on Real-time Computing and Robotics (RCAR).

[55]  Leo Galway,et al.  Feature Extraction for Emotion Recognition and Modelling Using Neurophysiological Data , 2016, 2016 15th International Conference on Ubiquitous Computing and Communications and 2016 International Symposium on Cyberspace and Security (IUCC-CSS).