Fusion of sentiment analysis and emotion recognition to model the user's emotional state

In this paper we present a framework that combines two algorithms respectively developed for Sentiment Analysis and Emotion Recognition in users spoken utterances. We propose modeling the users emotional state by means of the fusion of the outputs generated by both algorithms. This process considers the probabilities assigned to the different emotions by both algorithms. The proposed framework can be integrated as an additional module in the architecture of a spoken dialog system, using the information generated as an additional input for the dialog manager to decide the next system response.

[1]  Walaa Medhat,et al.  Sentiment analysis algorithms and applications: A survey , 2014 .

[2]  Yuxiao Hu,et al.  Audio-Visual Spontaneous Emotion Recognition , 2007, Artifical Intelligence for Human Computing.

[3]  Ruili Wang,et al.  Ensemble methods for spoken emotion recognition in call-centres , 2007, Speech Commun..

[4]  Timothy W. Bickmore,et al.  Maintaining reality: Relational agents for antipsychotic medication adherence , 2010, Interact. Comput..

[5]  José Neves,et al.  Context-Aware Emotion-Based Model for Group Decision Making , 2010, IEEE Intelligent Systems.

[6]  Carlos Ramos,et al.  Personality, Emotion, and Mood in Agent-Based Group Decision Making , 2011, IEEE Intelligent Systems.

[7]  Wolfgang Wahlster,et al.  Verbmobil: Foundations of Speech-to-Speech Translation , 2000, Artificial Intelligence.

[8]  Wolfgang Minker,et al.  Emotion recognition and adaptation in spoken dialogue systems , 2010, Int. J. Speech Technol..

[9]  Erik Cambria,et al.  The Hourglass of Emotions , 2011, COST 2102 Training School.

[10]  Angeliki Metallinou,et al.  Audio-Visual Emotion Recognition Using Gaussian Mixture Models for Face and Voice , 2008, 2008 Tenth IEEE International Symposium on Multimedia.

[11]  Mitsuru Ishizuka,et al.  Mandarin emotion recognition combining acoustic and emotional point information , 2012, Applied Intelligence.

[12]  Rada Mihalcea,et al.  Computational approaches to subjectivity and sentiment analysis: Present and envisaged methods and applications , 2014, Comput. Speech Lang..

[13]  Shrikanth S. Narayanan,et al.  Toward detecting emotions in spoken dialogs , 2005, IEEE Transactions on Speech and Audio Processing.

[14]  Ling Guan,et al.  Audiovisual emotion recognition via cross-modal association in kernel space , 2011, 2011 IEEE International Conference on Multimedia and Expo.

[15]  José Rouillard.  Web services and speech-based applications around VoiceXML , 2007, J. Networks.

[16]  Constantine Kotropoulos,et al.  Emotional speech recognition: Resources, features, and methods , 2006, Speech Commun..

[17]  Ramón López-Cózar,et al.  Relations between de-facto criteria in the evaluation of a spoken dialogue system , 2008, Speech Commun..

[18]  Wojtek Kowalczyk,et al.  Detecting changing emotions in human speech by machine and humans , 2013, Applied Intelligence.

[19]  Wolfgang Wahlster,et al.  SmartKom: Foundations of Multimodal Dialogue Systems (Cognitive Technologies) , 2006 .

[20]  Tim Polzehl,et al.  Emotion detection in dialog systems: Applications, strategies and challenges , 2009, 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops.

[21]  Ramón López-Cózar,et al.  Influence of contextual information in emotion annotation for spoken dialogue systems , 2008, Speech Commun..

[22]  John H. L. Hansen,et al.  Getting started with SUSAS: a speech under simulated and actual stress database , 1997, EUROSPEECH.

[23]  R. Plutchik,et al.  Emotion: Theory, Research, and Experience. Vol. 1. Theories of Emotion , 1981 .

[24]  Teruko Mitamura,et al.  Context-aware Dialog Strategies for Multimodal Mobile Dialog Systems , 2006 .

[25]  James R. Lewis.  The Voice in the Machine: Building Computers That Understand Speech , 2012, Int. J. Hum. Comput. Interact..

[26]  Ling Guan,et al.  Multimodal Information Fusion of Audio Emotion Recognition Based on Kernel Entropy Component Analysis , 2013, Int. J. Semantic Comput..

[27]  Björn W. Schuller,et al.  Recognising realistic emotions and affect in speech: State of the art and lessons learnt from the first challenge , 2011, Speech Commun..

[28]  Fakhri Karray,et al.  Survey on speech emotion recognition: Features, classification schemes, and databases , 2011, Pattern Recognit..

[29]  James R. Glass,et al.  Exploiting Context Information in Spoken Dialogue Interaction with Mobile Devices ? , 2022 .

[30]  Yorick Wilks,et al.  Some background on dialogue management and conversational speech for dialogue systems , 2011, Comput. Speech Lang..

[31]  Anton Nijholt,et al.  A tractable hybrid DDN–POMDP approach to affective dialogue modeling for probabilistic frame-based dialogue systems , 2008, Natural Language Engineering.

[32]  John H. L. Hansen,et al.  Analysis and compensation of speech under stress and noise for environmental robustness in speech recognition , 1996, Speech Commun..

[33]  Roberto Pieraccini,et al.  Automating spoken dialogue management design using machine learning: An industry perspective , 2008, Speech Commun..