Exploring Physiological Multimodality for Emotional Assessment

Many emotion recognition schemes have been proposed in the state-of-the-art. They generally differ in terms of the emotion elicitation methods, target emotional states to recognize, data sources or modalities, and classification techniques. In this work several biosignals are explored for emotion assessment during immersive video visualization, collecting multimodal data from Electrocardiography (ECG), Electrodermal Activity (EDA), Blood Volume Pulse (BVP) and Respiration sensors. Participants reported their emotional state of the day (baseline), and provided self-assessment of the emotion experienced in each video through the Self-Assessment Manikin (SAM), in the valence-arousal space. Multiple physiological and statistical features extracted from the biosignals were used as input to an emotion recognition workflow, targeting both user-dependent and user-independent classifications with three and two classes per dimension, respectively. Support vector machines (SVM) were used, as it is considered one of the most promising classifiers in the field. The proposed approach led to accuracies of 51.07% for arousal and 67.68% for valence in the user-dependent approach, and 69.13% for arousal and 67.75% for valence in the user-independent approach, which are encouraging for further research with a larger training dataset and population.

[1]  Eduardo Coutinho,et al.  Enhanced semi-supervised learning for multimodal emotion recognition , 2016, 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[2]  Nancy Alvarado.  Arousal and Valence in the Direct Scaling of Emotional Response to Film Clips , 1997 .

[3]  Chih-Jen Lin,et al.  A Practical Guide to Support Vector Classication , 2008 .

[4]  Johannes Wagner,et al.  From Physiological Signals to Emotions: Implementing and Comparing Selected Methods for Feature Extraction and Classification , 2005, 2005 IEEE International Conference on Multimedia and Expo.

[5]  Kemalasari,et al.  Processing of respiration signals using FIR filter for analyze the condition of lung , 2017, 2017 International Electronics Symposium on Engineering Technology and Applications (IES-ETA).

[6]  S. A. Hosseini,et al.  Classification of Brain Activity in Emotional States Using HOS Analysis , 2012 .

[7]  S. Veni,et al.  A multimodal emotion recognition system from video , 2016, 2016 International Conference on Circuit, Power and Computing Technologies (ICCPCT).

[8]  Mohammad Soleymani,et al.  Short-term emotion assessment in a recall paradigm , 2009, Int. J. Hum. Comput. Stud..

[9]  Goran Nenadic,et al.  Emotion Classification by Machine Learning Algorithm using Physiological Signals , .

[10]  Yoshua Bengio,et al.  Learning deep physiological models of affect , 2013, IEEE Computational Intelligence Magazine.

[11]  Crystal L. Hoyt,et al.  Immersive Virtual Environment Technology as a Methodological Tool for Social Psychology , 2002 .

[12]  Emre Ertin,et al.  mConverse: inferring conversation episodes from respiratory measurements collected in the field , 2011, Wireless Health.

[13]  J. Annett.  Subjective rating scales: science or art? , 2002, Ergonomics.

[14]  Henrik M. Peperkorn,et al.  The impact of perception and presence on emotional reactions: a review of research in virtual reality , 2015, Front. Psychol..

[15]  Ana L. N. Fred,et al.  Biosignals for Everyone , 2014, IEEE Pervasive Computing.

[16]  Murat Akcakaya,et al.  Artifact detection in electrodermal activity using sparse recovery , 2017, Commercial + Scientific Sensing and Imaging.

[17]  Vladimir Vapnik,et al.  An overview of statistical learning theory , 1999, IEEE Trans. Neural Networks.

[18]  Changchun Liu,et al.  An empirical study of machine learning techniques for affect recognition in human–robot interaction , 2006, 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[19]  Byoung-Jun Park,et al.  Analysis of physiological signals for recognition of boredom, pain, and surprise emotions , 2015, Journal of Physiological Anthropology.

[20]  E. Jang,et al.  Identification of the optimal emotion recognition algorithm using physiological signals , 2012, 2011 2nd International Conference on Engineering and Industries (ICEI).

[21]  P. Lang.  The emotion probe. Studies of motivation and attention. , 1995, The American psychologist.

[22]  Nicu Sebe,et al.  A Quality Adaptive Multimodal Affect Recognition System for User-Centric Multimedia Indexing , 2016, ICMR.

[23]  Jason Williams,et al.  Emotion Recognition Using Bio-sensors: First Steps towards an Automatic System , 2004, ADS.

[24]  E. Scilingo,et al.  Modeling for the Analysis of the EDA , 2016 .

[25]  T. Eerola,et al.  A comparison of the discrete and dimensional models of emotion in music , 2011 .

[26]  Marta S. N. M. Santos,et al.  Biometrical and Psychophysiological Assessment through Biosensors , 2012 .

[27]  Andreas Christmann,et al.  Support vector machines , 2008, Data Mining and Knowledge Discovery Handbook.

[28]  Xue Wang,et al.  Reputation-driven multimodal emotion recognition in wearable biosensor network , 2015, 2015 IEEE International Instrumentation and Measurement Technology Conference (I2MTC) Proceedings.

[29]  Mariano Alcañiz Raya,et al.  Affective Interactions Using Virtual Reality: The Link between Presence and Emotions , 2007, Cyberpsychology Behav. Soc. Netw..

[30]  M Murugappan,et al.  Physiological signals based human emotion Recognition: a review , 2011, 2011 IEEE 7th International Colloquium on Signal Processing and its Applications.

[31]  P. Hamilton,et al.  Open source ECG analysis , 2002, Computers in Cardiology.

[32]  Ana L. N. Fred,et al.  Outlier Detection in Non-intrusive ECG Biometric System , 2013, ICIAR.

[33]  Zongkai Yang,et al.  The study of emotion recognition from physiological signals , 2015, 2015 Seventh International Conference on Advanced Computational Intelligence (ICACI).

[34]  M. Murugappan,et al.  Classification of emotional states from electrocardiogram signals: a non-linear approach based on hurst , 2013, BioMedical Engineering OnLine.

[35]  Hauke R. Heekeren,et al.  Conceptualizing Emotions Along the Dimensions of Valence, Arousal, and Communicative Frequency – Implications for Social-Cognitive Tests and Training Tools , 2011, Front. Psychology.

[36]  P. Lang.  Behavioral treatment and bio-behavioral assessment: computer applications , 1980 .

[37]  Bong Siao Zheng,et al.  Frequency Band Analysis of Electrocardiogram (ECG) Signals for Human Emotional State Classification Using Discrete Wavelet Transform (DWT) , 2013, Journal of physical therapy science.

[38]  Roger G. Mark,et al.  An open-source algorithm to detect onset of arterial blood pressure pulses , 2003, Computers in Cardiology, 2003.

[39]  Jeremy N. Bailenson,et al.  A Public Database of Immersive VR Videos with Corresponding Ratings of Arousal, Valence, and Correlations between Head Movements and Self Report Measures , 2017, Front. Psychol..

[40]  Benoit Huet,et al.  Evidence Theory-Based Multimodal Emotion Recognition , 2009, MMM.

[41]  Bernhard E. Boser,et al.  A training algorithm for optimal margin classifiers , 1992, COLT '92.

[42]  Kyandoghere Kyamakya,et al.  Emotion Recognition Involving Physiological and Speech Signals: A Comprehensive Review , 2018 .

[43]  Christos D. Katsis,et al.  A User Independent, Biosignal Based, Emotion Recognition Method , 2007, User Modeling.

[44]  Neil Cooke,et al.  Multimodal affect recognition in an interactive gaming environment using eye tracking and speech signals , 2017, ICMI.

[45]  K. H. Kim,et al.  Emotion recognition system using short-term monitoring of physiological signals , 2004, Medical and Biological Engineering and Computing.

[46]  T. Dalgleish.  Basic Emotions , 2004 .

[47]  Ana L. N. Fred,et al.  Feature extraction for psychophysiological load assessment in unconstrained scenarios , 2012, 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[48]  Hillary Anger Elfenbein,et al.  On the universality and cultural specificity of emotion recognition: a meta-analysis. , 2002, Psychological bulletin.