Using unlabeled data to improve classification of emotional states in human computer interaction

The individual nature of physiological measurements of human affective states makes it very difficult to transfer statistical classifiers from one subject to another. In this work, we propose an approach to incorporate unlabeled data into a supervised classifier training in order to conduct an emotion classification. The key idea of the method is to conduct a density estimation of all available data (labeled and unlabeled) to create a new encoding of the problem. Based on this a supervised classifier is constructed. Further, numerical evaluations on the EmoRec II corpus are given, examining to what extent additional data can improve classification and which parameters of the density estimation are optimal.

[1]  J. Russell A circumplex model of affect. , 1980 .

[2]  Avrim Blum,et al.  The Bottleneck , 2021, Monopsony Capitalism.

[3]  Mohammad Soleymani,et al.  Affective Characterization of Movie Scenes Based on Multimedia Content Analysis and User's Physiological Emotional Responses , 2008, 2008 Tenth IEEE International Symposium on Multimedia.

[4]  Elisabeth André,et al.  Emotion recognition based on physiological changes in music listening , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[5]  Christine L. Lisetti,et al.  Using Noninvasive Wearable Computers to Recognize Human Emotions from Physiological Signals , 2004, EURASIP J. Adv. Signal Process..

[6]  G. Breithardt,et al.  Heart rate variability: standards of measurement, physiological interpretation and clinical use. Task Force of the European Society of Cardiology and the North American Society of Pacing and Electrophysiology. , 1996 .

[7]  Suzanne Kieffer,et al.  Feature extraction and selection for objective gait analysis and fall risk assessment by accelerometry , 2011, Biomedical engineering online.

[8]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[9]  T. Scherer,et al.  Constraints for emotion specificity in fear and anger: the context counts. , 2001, Psychophysiology.

[10]  Friedhelm Schwenker,et al.  Three learning phases for radial-basis-function networks , 2001, Neural Networks.

[11]  Anton van Boxtel,et al.  Facial EMG as a tool for inferring affective states , 2010 .

[12]  A. Malliani,et al.  Heart rate variability. Standards of measurement, physiological interpretation, and clinical use , 1996 .

[13]  Björn W. Schuller,et al.  AVEC 2011-The First International Audio/Visual Emotion Challenge , 2011, ACII.

[14]  J. C. Dill,et al.  Blood pressure responses and incentive appraisals as a function of perceived ability and objective task demand. , 1993, Psychophysiology.

[15]  Thomas M. Cover,et al.  Geometrical and Statistical Properties of Systems of Linear Inequalities with Applications in Pattern Recognition , 1965, IEEE Trans. Electron. Comput..

[16]  A. Mehrabian Pleasure-arousal-dominance: A general framework for describing and measuring individual differences in Temperament , 1996 .

[17]  Markus Kächele,et al.  Classification of Emotional States in a Woz Scenario Exploiting Labeled and Unlabeled Bio-physiological Data , 2011, PSL.

[18]  Pawel Strumillo,et al.  A Real-Time Adaptive Wavelet Transform-Based QRS Complex Detector , 2007, ICANNGA.

[19]  S M Pincus,et al.  Approximate entropy as a measure of system complexity. , 1991, Proceedings of the National Academy of Sciences of the United States of America.

[20]  Ludmila I. Kuncheva,et al.  Measures of Diversity in Classifier Ensembles and Their Relationship with the Ensemble Accuracy , 2003, Machine Learning.

[21]  M. Simson Use of Signals in the Terminal QRS Complex to Identify Patients with Ventricular Tachycardia After Myocardial Infarction , 1981, Circulation.

[22]  Elisabeth André,et al.  Exploring the benefits of discretization of acoustic features for speech emotion recognition , 2009, INTERSPEECH.

[23]  G. Stemmler,et al.  The autonomic differentiation of emotions revisited: convergent and discriminant validation. , 1989, Psychophysiology.

[24]  J. F. Kelley,et al.  An empirical methodology for writing user-friendly natural language computer applications , 1983, CHI '83.

[25]  E. Mugnaini,et al.  Cell junctions and intramembrane particles of astrocytes and oligodendrocytes: A freeze-fracture study , 1982, Neuroscience.

[26]  Björn W. Schuller,et al.  Active Learning by Sparse Instance Tracking and Classifier Confidence in Acoustic Emotion Recognition , 2012, INTERSPEECH.

[27]  Jennifer Healey,et al.  Toward Machine Emotional Intelligence: Analysis of Affective Physiological State , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[28]  J. Richman,et al.  Physiological time-series analysis using approximate entropy and sample entropy. , 2000, American journal of physiology. Heart and circulatory physiology.

[29]  M. Hoher,et al.  Adaptive class-specific partitioning as a means of initializing RBF-networks , 1995, 1995 IEEE International Conference on Systems, Man and Cybernetics. Intelligent Systems for the 21st Century.

[30]  D. Krikler,et al.  The QRS Complex , 1990, Annals of the New York Academy of Sciences.

[31]  Dietmar F. Rösner,et al.  LAST MINUTE: a Multimodal Corpus of Speech-based User-Companion Interactions , 2012, LREC.

[32]  Katherine B. Martin,et al.  Facial Action Coding System , 2015 .

[33]  Shaoning Pang,et al.  Transductive support vector machines and applications in bioinformatics for promoter recognition , 2003, International Conference on Neural Networks and Signal Processing, 2003. Proceedings of the 2003.

[34]  G. Rees,et al.  Neuroimaging: Decoding mental states from brain activity in humans , 2006, Nature Reviews Neuroscience.

[35]  Takeo Kanade,et al.  Comprehensive database for facial expression analysis , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[36]  J. Fahrenberg,et al.  Covariation and consistency of activation parameters , 1982, Biological Psychology.

[37]  Astrid Paeschke,et al.  A database of German emotional speech , 2005, INTERSPEECH.

[38]  H. Christensen,et al.  Power spectrum and turns analysis of EMG at different voluntary efforts in normal subjects. , 1986, Electroencephalography and clinical neurophysiology.

[39]  Shumeet Baluja,et al.  Probabilistic Modeling for Face Orientation Discrimination: Learning from Labeled and Unlabeled Data , 1998, NIPS.

[40]  Björn W. Schuller,et al.  Confidence Measures in Speech Emotion Recognition Based on Semi-supervised Learning , 2012, INTERSPEECH.

[41]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[42]  Martial Hebert,et al.  Semi-Supervised Self-Training of Object Detection Models , 2005, 2005 Seventh IEEE Workshops on Applications of Computer Vision (WACV/MOTION'05) - Volume 1.

[43]  B. Sayers,et al.  Analysis of heart rate variability. , 1973, Ergonomics.

[44]  N. Frijda,et al.  Emotions and respiratory patterns: review and critical analysis. , 1994, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[45]  Marimuthu Palaniswami,et al.  Sensitivity of temporal heart rate variability in Poincaré plot to changes in parasympathetic nervous system activity , 2011, Biomedical engineering online.

[46]  Jonghwa Kim,et al.  Transsituational Individual-Specific Biopsychological Classification of Emotions , 2013, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[47]  K. Scherer What are emotions? And how can they be measured? , 2005 .

[48]  M. Bradley,et al.  Looking at pictures: affective, facial, visceral, and behavioral reactions. , 1993, Psychophysiology.

[49]  Björn W. Schuller,et al.  Co-training succeeds in Computational Paralinguistics , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[50]  Karen J. Reynolds,et al.  Recurrence plot features of ECG signals , 1999, Proceedings of the First Joint BMES/EMBS Conference. 1999 IEEE Engineering in Medicine and Biology 21st Annual Conference and the 1999 Annual Fall Meeting of the Biomedical Engineering Society (Cat. N.

[51]  Günther Palm,et al.  The PIT Corpus of German Multi-Party Dialogues , 2008, LREC.

[52]  D. Ruelle,et al.  Recurrence Plots of Dynamical Systems , 1987 .

[53]  Friedhelm Schwenker,et al.  Studying Self- and Active-Training Methods for Multi-feature Set Emotion Recognition , 2011, PSL.

[54]  David A. Cohn,et al.  Active Learning with Statistical Models , 1996, NIPS.

[55]  Mohammad Soleymani,et al.  Queries and tags in affect-based multimedia retrieval , 2009, 2009 IEEE International Conference on Multimedia and Expo.

[56]  A W Frey,et al.  [Analysis of heart rate variability. Background, method, and possible use in anesthesia]. , 1995, Der Anaesthesist.

[57]  Thorsten Joachims,et al.  Transductive Support Vector Machines , 2006, Semi-Supervised Learning.

[58]  G. Qiu Indexing chromatic and achromatic patterns for content-based colour image retrieval , 2002, Pattern Recognit..

[59]  David B. Cooper,et al.  On the Asymptotic Improvement in the Out- come of Supervised Learning Provided by Additional Nonsupervised Learning , 1970, IEEE Transactions on Computers.

[60]  I. van Mechelen,et al.  Individual differences in core affect variability and their relationship to personality and psychological adjustment. , 2007, Emotion.

[61]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[62]  E. Vesterinen,et al.  Affective Computing , 2009, Encyclopedia of Biometrics.