Toward User-Independent Emotion Recognition Using Physiological Signals

Many techniques have been developed to improve the flexibility and the fit of detection models beyond user-dependent models, yet detection tasks continue to be complex and challenging. For emotion, which is known to be highly user-dependent, improvements to the emotion learning algorithm can greatly boost predictive power. Our aim is to improve the accuracy rate of the classifier using peripheral physiological signals. Here, we present a hybrid sensor fusion approach based on a stacking model that allows for data from multiple sensors and emotion models to be jointly embedded within a user-independent model. WMD-DTW, which is a weighted multi-dimensional DTW, and the k-nearest neighbor’s algorithm k-NN are used to classify the emotions as a base model. The ensemble methods were used to learn a high-level classifier on top of the two base models. We applied a meta-learning approach to the data set and showed that the ensemble approach outperforms any individual method. We also compared the results using two data sets. Our proposed system achieved an overall accuracies of 65.6% for all users for the E4-data set and 94.0% and 93.6% for recognizing valence and arousal emotions, respectively, using the MAHNOB data set.

[1]  Luca Citi,et al.  Skin Admittance Measurement for Emotion Recognition: A Study over Frequency Sweep , 2016 .

[2]  Jun Wang,et al.  Generalizing DTW to the multi-dimensional case requires an adaptive approach , 2016, Data Mining and Knowledge Discovery.

[3]  K. H. Kim,et al.  Emotion recognition system using short-term monitoring of physiological signals , 2004, Medical and Biological Engineering and Computing.

[4]  Enzo Pasquale Scilingo,et al.  Improving emotion recognition systems by embedding cardiorespiratory coupling , 2013, Physiological measurement.

[5]  Steven M. LaValle,et al.  On the Relationship between Classical Grid Search and Probabilistic Roadmaps , 2004, Int. J. Robotics Res..

[6]  Turhan Canli,et al.  Individual differences in emotion processing , 2004, Current Opinion in Neurobiology.

[7]  Francis Tuerlinckx,et al.  The Relation Between Valence and Arousal in Subjective Experience Varies With Personality and Culture. , 2017, Journal of personality.

[8]  P. Ekman Are there basic emotions? , 1992, Psychological review.

[9]  Ron Kohavi,et al.  A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection , 1995, IJCAI.

[10]  Jun Wang,et al.  Sentence recognition from articulatory movements for silent speech interfaces , 2012, 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[11]  Kostas Karpouzis,et al.  The HUMAINE Database: Addressing the Collection and Annotation of Naturalistic and Induced Emotional Data , 2007, ACII.

[12]  Michael G. Madden,et al.  An Ensemble Dynamic Time Warping Classifier with Application to Activity Recognition , 2010, SGAI Conf..

[13]  Ian H. Witten,et al.  Issues in Stacked Generalization , 2011, J. Artif. Intell. Res..

[14]  J. Russell,et al.  A 12-Point Circumplex Structure of Core Affect. , 2011, Emotion.

[15]  William J. Emery,et al.  Data Analysis Methods in Physical Oceanography , 1998 .

[16]  C. E. Izard Organizational and motivational functions of discrete emotions. , 1993 .

[17]  M. Reinders,et al.  Multi-Dimensional Dynamic Time Warping for Gesture Recognition , 2007 .

[18]  Tarik Arici,et al.  Robust gesture recognition using feature pre-processing and weighted dynamic time warping , 2014, Multimedia Tools and Applications.

[19]  Fadel Adib,et al.  Emotion recognition using wireless signals , 2016, MobiCom.

[20]  Su-Lim Tan,et al.  A biometric signature based system for improved emotion recognition using physiological responses from multiple subjects , 2010, 2010 8th IEEE International Conference on Industrial Informatics.

[21]  Francis Tuerlinckx,et al.  Predicting the duration of emotional experience: two experience sampling studies. , 2009, Emotion.

[22]  B. T. Engel Stimulus-Response and Individual-Response Specificity , 1960 .

[23]  Nando de Freitas,et al.  Taking the Human Out of the Loop: A Review of Bayesian Optimization , 2016, Proceedings of the IEEE.

[24]  David Sander,et al.  A systems approach to appraisal mechanisms in emotion , 2005, Neural Networks.

[25]  Jasper Snoek,et al.  Practical Bayesian Optimization of Machine Learning Algorithms , 2012, NIPS.

[26]  M. Shamim Hossain,et al.  Audio-visual emotion recognition using multi-directional regression and Ridgelet transform , 2016, Journal on Multimodal User Interfaces.

[27]  Mohammad Soleymani,et al.  A Multimodal Database for Affect Recognition and Implicit Tagging , 2012, IEEE Transactions on Affective Computing.

[28]  J. Russell,et al.  A Description of the Affective Quality Attributed to Environments , 1980 .

[29]  J. Russell Core affect and the psychological construction of emotion. , 2003, Psychological review.

[30]  L. F. Barrett Are Emotions Natural Kinds? , 2006, Perspectives on psychological science : a journal of the Association for Psychological Science.

[31]  P. Totterdell,et al.  Forecasting Feelings: The accuracy and effects of self-predictions of mood , 1997 .

[32]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[33]  J. Gross,et al.  The tie that binds? Coherence among emotion experience, behavior, and physiology. , 2005, Emotion.

[34]  J. E. Rose,et al.  Autonomic Nervous System Activity Distinguishes Among Emotions , 2009 .

[35]  Zied Lachiri,et al.  Emotion Classification in Arousal Valence Model using MAHNOB-HCI Database , 2017 .

[36]  Rosalind W. Picard,et al.  Empatica E3 — A wearable wireless multi-sensor device for real-time computerized biofeedback and data acquisition , 2014, 2014 4th International Conference on Wireless Mobile Communication and Healthcare - Transforming Healthcare Through Innovations in Mobile and Wireless Technologies (MOBIHEALTH).

[37]  Shane S. Sturrock,et al.  Time Warps, String Edits, and Macromolecules – The Theory and Practice of Sequence Comparison . David Sankoff and Joseph Kruskal. ISBN 1-57586-217-4. Price £13.95 (US$22·95). , 2000 .

[38]  K. Poels,et al.  How to Capture the Heart? Reviewing 20 Years of Emotion Measurement in Advertising , 2006 .

[39]  Hassan Ghasemzadeh,et al.  Multi-sensor fusion in body sensor networks: State-of-the-art and research challenges , 2017, Inf. Fusion.

[40]  Bertrand Clarke,et al.  Comparing Bayes Model Averaging and Stacking When Model Approximation Error Cannot be Ignored , 2003, J. Mach. Learn. Res..

[41]  Fabien Ringeval,et al.  End-to-end learning for dimensional emotion recognition from physiological signals , 2017, 2017 IEEE International Conference on Multimedia and Expo (ICME).

[42]  François Petitjean,et al.  Satellite Image Time Series Analysis Under Time Warping , 2014, IEEE Transactions on Geoscience and Remote Sensing.

[43]  J. Russell,et al.  Distinguishing anger and anxiety in terms of emotional response factors. , 1974, Journal of consulting and clinical psychology.

[44]  Elisabeth André,et al.  Emotion recognition based on physiological changes in music listening , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[45]  C. Maaoui,et al.  Emotion Recognition through Physiological Signals for Human-Machine Communication , 2010 .

[46]  G. Stemmler,et al.  Personality, emotion, and individual differences in physiological responses , 2010, Biological Psychology.

[47]  Jennifer Healey,et al.  Detecting stress during real-world driving tasks using physiological sensors , 2005, IEEE Transactions on Intelligent Transportation Systems.

[48]  Christos D. Katsis,et al.  Toward Emotion Recognition in Car-Racing Drivers: A Biosignal Processing Approach , 2008, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[49]  M. King,et al.  Social desirability bias: A neglected aspect of validity testing , 2000 .

[50]  Muhammad Ghulam,et al.  User emotion recognition from a larger pool of social network data using active learning , 2017, Multimedia Tools and Applications.

[51]  Yoshua Bengio,et al.  Algorithms for Hyper-Parameter Optimization , 2011, NIPS.

[52]  Jean-Philippe Thiran,et al.  Prediction of asynchronous dimensional emotion ratings from audiovisual and physiological data , 2015, Pattern Recognit. Lett..

[53]  David H. Wolpert,et al.  Stacked generalization , 1992, Neural Networks.

[54]  Ó. Gonçalves,et al.  The Emotional Movie Database (EMDB): A Self-Report and Psychophysiological Study , 2012, Applied psychophysiology and biofeedback.

[55]  Olufemi A. Omitaomu,et al.  Weighted dynamic time warping for time series classification , 2011, Pattern Recognit..

[56]  Charu C. Aggarwal,et al.  Feature Selection for Classification: A Review , 2014, Data Classification: Algorithms and Applications.

[57]  I. van Mechelen,et al.  Individual differences in core affect variability and their relationship to personality and psychological adjustment. , 2007, Emotion.

[58]  Jürgen S. Sauer,et al.  AniSAM & AniAvatar: Animated Visualizations of Affective States , 2016, CHI.

[59]  M.H. Moradi,et al.  Emotion detection using brain and peripheral signals , 2008, 2008 Cairo International Biomedical Engineering Conference.

[60]  Marc D. Lewis Bridging emotion theory and neurobiology through dynamic systems modeling , 2005, Behavioral and Brain Sciences.

[61]  Thierry Pun,et al.  DEAP: A Database for Emotion Analysis ;Using Physiological Signals , 2012, IEEE Transactions on Affective Computing.

[62]  Chao Li,et al.  Analysis of physiological for emotion recognition with the IRS model , 2016, Neurocomputing.

[63]  Stéphane Bonnet,et al.  Selection of the Most Relevant Physiological Features for Classifying Emotion , 2015, PhyCS.

[64]  Giovanni Seni,et al.  Ensemble Methods in Data Mining: Improving Accuracy Through Combining Predictions , 2010, Ensemble Methods in Data Mining.

[65]  Min Chen,et al.  Emotion Communication System , 2017, IEEE Access.

[66]  Peter Kuppens,et al.  Individual differences in emotion components and dynamics: Introduction to the Special Issue , 2009 .

[67]  R. Cabeza,et al.  Cognitive neuroscience of emotional memory , 2006, Nature Reviews Neuroscience.

[68]  Gary P. Pisano,et al.  Toward a Theory of Behavioral Operations , 2008, Manuf. Serv. Oper. Manag..