Emotion detection from text and speech: a survey

Emotion recognition has emerged as an important research area which may reveal some valuable input to a variety of purposes. People express their emotions directly or indirectly through their speech, facial expressions, gestures or writings. Many different sources of information, such as speech, text and visual can be used to analyze emotions. Nowadays, writings take many forms of social media posts, micro-blogs, news articles, etc., and the content of these posts can be useful resource for text mining to discover and unhide various aspects, including emotions. Extracting emotions behind these postings is an immense and complicated task. To tackle this problem, researchers from diverse fields are trying to find an efficient way to more precisely detect human emotions from various sources, including text and speech. In this sense, different word-based and sentence-based techniques, machine learning, natural language processing methods, etc., have been used to achieve better accuracy. Analyzing emotions can be helpful in many different domains. One such domain is human computer interaction. With the help of emotion recognition, computers can make better decisions to help users. With the increase in popularity of robotic research, emotion recognition will also help making human–robot interaction more natural. This survey covers existing emotion detection research efforts, emotion models, emotion datasets, emotion detection techniques, their features, limitations and some possible future directions. We focus on reviewing research efforts analyzing emotions based on text and speech. We investigated different feature sets that have been used in existing methodologies. We summarize basic achievements in the field and highlight possible extensions for better outcome.

[1]  Shrikanth Narayanan,et al.  Recognition of negative emotions from the speech signal , 2001, IEEE Workshop on Automatic Speech Recognition and Understanding, 2001. ASRU '01..

[2]  Kornel Laskowski,et al.  Emotion recognition in spontaneous speech using GMMs , 2006, INTERSPEECH.

[3]  Gang Wei,et al.  Speech emotion recognition based on HMM and SVM , 2005, 2005 International Conference on Machine Learning and Cybernetics.

[4]  Tsutomu Miyasato,et al.  Emotion recognition from audiovisual information , 1998, 1998 IEEE Second Workshop on Multimedia Signal Processing (Cat. No.98EX175).

[5]  Yi Luo,et al.  Speech emotion recognition based on a hybrid of HMM/ANN , 2007 .

[6]  Amandeep Verma,et al.  Design for emotion detection of punjabi text using hybrid approach , 2016, 2016 International Conference on Inventive Computation Technologies (ICICT).

[7]  Igor Kononenko,et al.  Estimating Attributes: Analysis and Extensions of RELIEF , 1994, ECML.

[8]  Alexander Maedche,et al.  Advanced User Assistance Systems , 2016, Bus. Inf. Syst. Eng..

[9]  Abhijeet Kumar,et al.  Automatic speech emotion detection system using multi-domain acoustic feature selection and classification models , 2017, 2017 IEEE International Conference on Identity, Security and Behavior Analysis (ISBA).

[10]  Chu-Ren Huang,et al.  A Text-driven Rule-based System for Emotion Cause Detection , 2010, HLT-NAACL 2010.

[11]  Fuji Ren,et al.  Exploring latent semantic information for textual emotion recognition in blog articles , 2018, IEEE/CAA Journal of Automatica Sinica.

[12]  Ioannis Hatzilygeroudis,et al.  Recognizing emotions in text using ensemble of classifiers , 2016, Eng. Appl. Artif. Intell..

[13]  Josef Kittler,et al.  Pattern recognition : a statistical approach , 1982 .

[14]  Shiqing Zhang,et al.  Speech Emotion Recognition Using Support Vector Machines , 2011, CSIE 2011.

[15]  H. Lövheim A new three-dimensional model for emotions and monoamine neurotransmitters. , 2012, Medical hypotheses.

[16]  H. Akaike,et al.  Information Theory and an Extension of the Maximum Likelihood Principle , 1973 .

[17]  M. Rosenblum,et al.  Human emotion recognition from motion using a radial basis function network architecture , 1994, Proceedings of 1994 IEEE Workshop on Motion of Non-rigid and Articulated Objects.

[18]  Aijun An,et al.  Unsupervised Emotion Detection from Text Using Semantic and Syntactic Relations , 2012, 2012 IEEE/WIC/ACM International Conferences on Web Intelligence and Intelligent Agent Technology.

[19]  Nikos A. Vlassis,et al.  A Greedy EM Algorithm for Gaussian Mixture Learning , 2002, Neural Processing Letters.

[20]  Von-Wun Soo,et al.  Towards Text-based Emotion Detection A Survey and Possible Improvements , 2009, 2009 International Conference on Information Management and Engineering.

[21]  Hua Xu,et al.  Text-based emotion classification using emotion cause extraction , 2014, Expert Syst. Appl..

[22]  Yixiong Pan,et al.  SPEECH EMOTION RECOGNITION USING SUPPORT VECTOR MACHINE , 2010 .

[23]  Lyle H. Ungar,et al.  Modelling Valence and Arousal in Facebook posts , 2016, WASSA@NAACL-HLT.

[24]  Chetan R. Chopade Based Emotion Recognition : A Survey , 2015 .

[25]  P. Johnson-Laird,et al.  Towards a Cognitive Theory of Emotions , 1987 .

[26]  Md. Kamrul Hasan,et al.  Identifying emotion by keystroke dynamics and text pattern analysis , 2014, Behav. Inf. Technol..

[27]  Sunghwan Mac Kim,et al.  EMOTIONS IN TEXT: DIMENSIONAL AND CATEGORICAL MODELS , 2013, Comput. Intell..

[28]  Osmar R. Zaïane,et al.  Current State of Text Sentiment Analysis from Opinion to Emotion Mining , 2017, ACM Comput. Surv..

[29]  J. Russell A circumplex model of affect. , 1980 .

[30]  Wei Wu,et al.  GMM Supervector Based SVM with Spectral Features for Speech Emotion Recognition , 2007, 2007 IEEE International Conference on Acoustics, Speech and Signal Processing - ICASSP '07.

[31]  J. Borod The Neuropsychology of emotion , 2000 .

[32]  Valery A. Petrushin,et al.  Emotion recognition in speech signal: experimental study, development, and application , 2000, INTERSPEECH.

[33]  C. D. Broad,et al.  Emotion and Sentiment , 1954 .

[34]  Luca Dini,et al.  Emotion Analysis on Twitter: The Hidden Challenge , 2016, LREC.

[35]  Shourya Roy,et al.  Multi-task Representation Learning for Enhanced Emotion Categorization in Short Text , 2017, PAKDD.

[36]  Hajar Mousannif,et al.  Using YouTube Comments for Text-based Emotion Recognition , 2016, ANT/SEIT.

[37]  Véronique Hoste,et al.  Emotion detection in suicide notes , 2013, Expert Syst. Appl..

[38]  Shikha Tripathi,et al.  Emotion Recognition through Speech Signal for Human-Computer Interaction , 2014, 2014 Fifth International Symposium on Electronic System Design.

[39]  Jerome R. Bellegarda,et al.  Large-scale personal assistant technology deployment: the siri experience , 2013, INTERSPEECH.

[40]  Yanghui Rao,et al.  Hybrid neural networks for social emotion detection over short text , 2016, 2016 International Joint Conference on Neural Networks (IJCNN).

[41]  Douiji yasmina,et al.  Using YouTube Comments for Text-based Emotion Recognition☆ , 2016 .

[42]  Loïc Kessous,et al.  Multimodal emotion recognition in speech-based interaction using facial expression, body gesture and acoustic analysis , 2010, Journal on Multimodal User Interfaces.

[43]  Mohammad Soleymani,et al.  Analysis of EEG Signals and Facial Expressions for Continuous Emotion Detection , 2016, IEEE Transactions on Affective Computing.

[44]  Erik Cambria,et al.  Fusing audio, visual and textual clues for sentiment analysis from multimodal content , 2016, Neurocomputing.

[45]  Prashant Lahane,et al.  Emotion Recognition from Facial Expressions using Multilevel HMM ( , 2014 .

[46]  Theodoros Iliou,et al.  Features and classifiers for emotion recognition from speech: a survey from 2000 to 2011 , 2012, Artificial Intelligence Review.

[47]  Dong Yu,et al.  Speech emotion recognition using deep neural network and extreme learning machine , 2014, INTERSPEECH.

[48]  P. Shaver,et al.  Emotion knowledge: further exploration of a prototype approach. , 1987, Journal of personality and social psychology.

[49]  C. Darwin The Expression of the Emotions in Man and Animals , .

[50]  Steven Lawrence Fernandes,et al.  Extraction of emotions from multilingual text using intelligent text processing and computational linguistics , 2017, J. Comput. Sci..

[51]  Shiv Naresh Shivhare,et al.  Emotion Detection from Text , 2012, ArXiv.

[52]  Christopher Joseph Pal,et al.  EmoNets: Multimodal deep learning approaches for emotion recognition in video , 2015, Journal on Multimodal User Interfaces.

[53]  Bernd Resch,et al.  Microblog Emotion Classification by Computing Similarity in Text, Time, and Space , 2016, PEOPLES@COLING.

[54]  S. Lalitha,et al.  Speech emotion recognition , 2014, 2014 International Conference on Advances in Electronics Computers and Communications.

[55]  Connor Graham,et al.  A Review of Mobile HCI Research Methods , 2003, Mobile HCI.

[56]  Puneet Agrawal,et al.  A Sentiment-and-Semantics-Based Approach for Emotion Detection in Textual Conversations , 2017, ArXiv.

[57]  P. Ekman An argument for basic emotions , 1992 .

[58]  Shujuan Wang,et al.  Naive Bayes classifiers for music emotion classification based on lyrics , 2017, 2017 IEEE/ACIS 16th International Conference on Computer and Information Science (ICIS).

[59]  Carlo Strapparava,et al.  Learning to identify emotions in text , 2008, SAC '08.

[60]  L. C. De Silva,et al.  Bimodal emotion recognition , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[61]  Elke A. Rundensteiner,et al.  EMOTEX: Detecting Emotions in Twitter Messages , 2014 .

[62]  Björn W. Schuller,et al.  Speech emotion recognition combining acoustic features and linguistic information in a hybrid support vector machine-belief network architecture , 2004, 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[63]  Elke A. Rundensteiner,et al.  Using Social Sensing to Discover Trends in Public Emotion , 2017, 2017 IEEE 11th International Conference on Semantic Computing (ICSC).

[64]  J. Sartre,et al.  Sketch for a Theory of the Emotions , 1939 .

[65]  Pushpak Bhattacharyya,et al.  Emotion Analysis from Text : A Survey , 2016 .

[66]  Vidyasagar Potdar,et al.  Computational approaches for emotion detection in text , 2010, 4th IEEE International Conference on Digital Ecosystems and Technologies.

[67]  Diana Inkpen,et al.  Prior and contextual emotion of words in sentential context , 2014, Comput. Speech Lang..

[68]  Thomas S. Huang,et al.  Emotion Recognition from Facial Expressions using Multilevel HMM , 2000 .

[69]  Patricio Martínez-Barco,et al.  Emotion Detection from text: A Survey , 2014 .

[70]  Fakhri Karray,et al.  Survey on speech emotion recognition: Features, classification schemes, and databases , 2011, Pattern Recognit..

[71]  Mann Oo. Hay Emotion recognition in human-computer interaction , 2012 .

[72]  S. Tiwari,et al.  A Novel Approach for Detecting Emotion in Text , 2016 .

[73]  Brad A. Myers,et al.  Using HCI techniques to design a more usable programming system , 2002, Proceedings IEEE 2002 Symposia on Human Centric Computing Languages and Environments.

[74]  Abir Hudait,et al.  Automatic emotion detection model from facial expression , 2016, 2016 International Conference on Advanced Communication Control and Computing Technologies (ICACCCT).

[75]  M. Yunus Nayan,et al.  Human emotion detection through speech and facial expressions , 2016, 2016 3rd International Conference on Computer and Information Sciences (ICCOINS).

[76]  Johannes Wagner,et al.  Emotional Sensitivity in Human-Computer Interaction , 2009, it Inf. Technol..

[77]  Anil Kumar,et al.  Performance Analysis of Reversible ALU in QCA , 2017 .

[78]  Jiucang Hao,et al.  Emotion recognition by speech signals , 2003, INTERSPEECH.

[79]  Mark Garland Hayden,et al.  The Ensemble System , 1998 .

[80]  Oh-Wook Kwon,et al.  EMOTION RECOGNITION BY SPEECH SIGNAL , 2003 .

[81]  Erik Cambria,et al.  Convolutional MKL Based Multimodal Emotion Recognition and Sentiment Analysis , 2016, 2016 IEEE 16th International Conference on Data Mining (ICDM).

[82]  V. Sharmila,et al.  Using Hashtags to Capture Fine Emotion Categories from Tweets , 2019 .

[83]  Klaus Willmes,et al.  Hemispheric Asymmetry in the Recognition of Emotional Attitude Conveyed by Facial Expression, Prosody and Propositional Speech , 1997, Cortex.

[84]  Pushpak Bhattacharyya,et al.  EmoGram: An Open-Source Time Sequence-Based Emotion Tracker and Its Innovative Applications , 2016, AAAI Workshop: Knowledge Extraction from Text.

[85]  R. Plutchik Emotion, a psychoevolutionary synthesis , 1980 .

[86]  Andrew Ortony,et al.  The Cognitive Structure of Emotions , 1988 .

[87]  Zhigang Deng,et al.  Analysis of emotion recognition using facial expressions, speech and multimodal information , 2004, ICMI '04.

[88]  M MohammadSaif,et al.  Using Hashtags to Capture Fine Emotion Categories from Tweets , 2015, CI 2015.

[89]  Gourav Bathla,et al.  Recognizing Emotion in Text using Neural Network and Fuzzy Logic , 2017 .

[90]  Erkki Sutinen,et al.  Are They Different? Affect, Feeling, Emotion, Sentiment, and Opinion Detection in Text , 2014, IEEE Transactions on Affective Computing.

[91]  Saif Mohammad,et al.  Emotion Intensities in Tweets , 2017, *SEMEVAL.