Deep learning for affective computing: Text-based emotion recognition in decision support

Abstract Emotions widely affect human decision-making. This fact is taken into account by affective computing with the goal of tailoring decision support to the emotional states of individuals. However, the accurate recognition of emotions within narrative documents presents a challenging undertaking due to the complexity and ambiguity of language. Performance improvements can be achieved through deep learning; yet, as demonstrated in this paper, the specific nature of this task requires the customization of recurrent neural networks with regard to bidirectional processing, dropout layers as a means of regularization, and weighted loss functions. In addition, we propose sent2affect, a tailored form of transfer learning for affective computing: here the network is pre-trained for a different task (i.e. sentiment analysis), while the output layer is subsequently tuned to the task of emotion recognition. The resulting performance is evaluated in a holistic setting across 6 benchmark datasets, where we find that both recurrent neural networks and transfer learning consistently outperform traditional machine learning. Altogether, the findings have considerable implications for the use of affective computing.

[1]  Puneet Agrawal,et al.  A Sentiment-and-Semantics-Based Approach for Emotion Detection in Textual Conversations , 2017, ArXiv.

[2]  Kaushal Kumar Shukla,et al.  Prayas at EmoInt 2017: An Ensemble of Deep Neural Architectures for Emotion Intensity Prediction in Tweets , 2017, WASSA@EMNLP.

[3]  Han Zhang,et al.  Anxious or Angry? Effects of Discrete Emotions on the Perceived Helpfulness of Online Reviews , 2014, MIS Q..

[4]  Stefan Feuerriegel,et al.  Putting Question-Answering Systems into Practice , 2018, Decis. Support Syst..

[5]  Joel D. Martin,et al.  Sentiment, emotion, purpose, and style in electoral tweets , 2015, Inf. Process. Manag..

[6]  Saif Mohammad,et al.  From once upon a time to happily ever after: Tracking emotions in mail and books , 2012, Decis. Support Syst..

[7]  Jella Pfeiffer,et al.  Using Live Biofeedback for Decision Support: Investigating Influences of Emotion Regulation in Financial Decision Making , 2015, ECIS.

[8]  Yen-Liang Chen,et al.  Emotion classification of YouTube videos , 2017, Decis. Support Syst..

[9]  Qiang Yang,et al.  A Survey on Transfer Learning , 2010, IEEE Transactions on Knowledge and Data Engineering.

[10]  Swee Hoon Ang,et al.  Exploring the dimensions of ad creativity , 2000 .

[11]  Shaogang Gong,et al.  Facial expression recognition based on Local Binary Patterns: A comprehensive study , 2009, Image Vis. Comput..

[12]  J. Averill A CONSTRUCTIVIST VIEW OF EMOTION , 1980 .

[13]  C. Natalie van der Wal,et al.  An Agent-Based Model Predicting Group Emotion and Misbehaviours in Stranded Passengers , 2017, EPIA.

[14]  Stefan Feuerriegel,et al.  Decision support from financial disclosures with deep neural networks and transfer learning , 2017, Decis. Support Syst..

[15]  Erik Cambria,et al.  The Hourglass of Emotions , 2011, COST 2102 Training School.

[16]  Rui Yan,et al.  How Transferable are Neural Networks in NLP Applications? , 2016, EMNLP.

[17]  Michael W. Eysenck,et al.  Introduction to the special issue: Emotional states, attention, and working memory , 2010 .

[18]  R. Dolan,et al.  Emotion, Cognition, and Behavior , 2002, Science.

[19]  Yoshua Bengio,et al.  Learning long-term dependencies with gradient descent is difficult , 1994, IEEE Trans. Neural Networks.

[20]  Aijun An,et al.  Unsupervised Emotion Detection from Text Using Semantic and Syntactic Relations , 2012, 2012 IEEE/WIC/ACM International Conferences on Web Intelligence and Intelligent Agent Technology.

[21]  Osmar R. Zaïane,et al.  Current State of Text Sentiment Analysis from Opinion to Emotion Mining , 2017, ACM Comput. Surv..

[22]  Gregory J. Park,et al.  Psychological Language on Twitter Predicts County-Level Heart Disease Mortality , 2015, Psychological science.

[23]  Uzay Kaymak,et al.  A Survey of event extraction methods from text for decision support systems , 2016, Decis. Support Syst..

[24]  Michael,et al.  Affective Computing and Intelligent Interaction , 2011, Lecture Notes in Computer Science.

[25]  Wonjoon Kim,et al.  From valence to emotions: Exploring the distribution of emotions in online product reviews , 2016, Decis. Support Syst..

[26]  Craig A. Smith,et al.  Cognition and Emotion over twenty-five years , 2011, Cognition & emotion.

[27]  Eric Gilbert,et al.  Widespread Worry and the Stock Market , 2010, ICWSM.

[28]  C. Izard Emotion theory and research: highlights, unanswered questions, and emerging issues. , 2009, Annual review of psychology.

[29]  Denis Gordeev,et al.  Detecting State of Aggression in Sentences Using CNN , 2016, SPECOM.

[30]  Saif Mohammad,et al.  CROWDSOURCING A WORD–EMOTION ASSOCIATION LEXICON , 2013, Comput. Intell..

[31]  Fabrício Benevenuto,et al.  A Benchmark Comparison of State-of-the-Practice Sentiment Analysis Methods , 2015, ArXiv.

[32]  A. Darzi,et al.  Harnessing the cloud of patient experience: using social media to detect poor quality healthcare , 2013, BMJ quality & safety.

[33]  Gary King,et al.  Logistic Regression in Rare Events Data , 2001, Political Analysis.

[34]  Yunfei Long,et al.  Inferring Affective Meanings of Words from Word Embedding , 2017, IEEE Transactions on Affective Computing.

[35]  R. Plutchik The Nature of Emotions , 2001 .

[36]  Fakhri Karray,et al.  Survey on speech emotion recognition: Features, classification schemes, and databases , 2011, Pattern Recognit..

[37]  P. Ekman,et al.  DIFFERENCES Universals and Cultural Differences in the Judgments of Facial Expressions of Emotion , 2004 .

[38]  E. Vesterinen,et al.  Affective Computing , 2009, Encyclopedia of Biometrics.

[39]  Chih-Jen Lin,et al.  A Practical Guide to Support Vector Classication , 2008 .

[40]  Stefan Wermter,et al.  GradAscent at EmoInt-2017: Character and Word Level Recurrent Neural Network Models for Tweet Emotion Intensity Detection , 2017, WASSA@EMNLP.

[41]  N. Frijda The laws of emotion. , 1988, The American psychologist.

[42]  V. Mahajan,et al.  Form versus Function: How the Intensities of Specific Emotions Evoked in Functional versus Hedonic Trade-Offs Mediate Product Preferences , 2007 .

[43]  Xuejie Zhang,et al.  YNU-HPCC at EmoInt-2017: Using a CNN-LSTM Model for Sentiment Intensity Prediction , 2017, WASSA@EMNLP.

[44]  R. Gorrell Affect, Imagery, and Consciousness: vol I. Positive Affects. , 1963 .

[45]  Stewart Massie,et al.  Lexicon based feature extraction for emotion text classification , 2017, Pattern Recognit. Lett..

[46]  Pablo Moscato,et al.  Deep neural networks understand investors better , 2018, Decis. Support Syst..

[47]  Rafael A. Calvo,et al.  Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications , 2010, IEEE Transactions on Affective Computing.

[48]  Jana-Rebecca Rehse,et al.  Predicting process behaviour using deep learning , 2016, Decis. Support Syst..

[49]  Véronique Hoste,et al.  Emotion detection in suicide notes , 2013, Expert Syst. Appl..

[50]  Matt E. Thatcher,et al.  The effects of positive affect and personal information search on outcomes in call centers: An empirical study , 2012, Decis. Support Syst..

[51]  Pilar Rodríguez Marín,et al.  Extracting Emotions from Texts in E-Learning Environments , 2012, 2012 Sixth International Conference on Complex, Intelligent, and Software Intensive Systems.

[52]  J. Pennebaker,et al.  The Psychological Meaning of Words: LIWC and Computerized Text Analysis Methods , 2010 .

[53]  P. Ormerod,et al.  Measuring Financial Sentiment to Predict Financial Instability: A New Approach based on Text Analysis , 2015, 1508.05357.

[54]  Cecilia Ovesdotter Alm,et al.  Affect in Text and Speech , 2009 .

[55]  A ValentinoNicholas,et al.  Election Night’s Alright for Fighting: The Role of Emotions in Political Participation , 2011 .

[56]  Lillian Lee,et al.  Opinion Mining and Sentiment Analysis , 2008, Found. Trends Inf. Retr..

[57]  Stefan Feuerriegel,et al.  Deep learning in business analytics and operations research: Models, applications and managerial implications , 2018, Eur. J. Oper. Res..

[58]  Rathindra Sarathy,et al.  The role of affect and cognition on online consumers' decision to disclose personal information to unfamiliar online vendors , 2011, Decis. Support Syst..

[59]  Peter V. Marsden,et al.  Reflections on Conceptualizing and Measuring Tie Strength , 2012 .

[60]  Ritu Agarwal,et al.  The Digitization of Healthcare: Boundary Risks, Emotion, and Consumer Willingness to Disclose Personal Health Information , 2011, Inf. Syst. Res..

[61]  J. Russell A circumplex model of affect. , 1980 .

[62]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[63]  Stefan Feuerriegel,et al.  Is Human Information Processing Affected by Emotional Content? Understanding The Role of Facts and Emotions in the Stock Market , 2016, ICIS.

[64]  Hardik Meisheri,et al.  Textmining at EmoInt-2017: A Deep Learning Approach to Sentiment Intensity Scoring of English Tweets , 2017, WASSA@EMNLP.

[65]  Pim Cuijpers,et al.  Web-based depression treatment: associations of clients' word use with adherence and outcome. , 2014, Journal of affective disorders.

[66]  T. Danisman,et al.  Feeler: Emotion Classification of Text Using Vector Space Model , 2008 .

[67]  Dua'a Al-Hajjar,et al.  Applying sentiment and emotion analysis on brand tweets for digital marketing , 2015, 2015 IEEE Jordan Conference on Applied Electrical Engineering and Computing Technologies (AEECT).

[68]  N. Schwarz Emotion, cognition, and decision making , 2000 .

[69]  Athena Vakali,et al.  Detecting variation of emotions in online activities , 2017, Expert Syst. Appl..

[70]  C. Izard Basic emotions, relations among emotions, and emotion-cognition relations. , 1992, Psychological review.

[71]  Joshua D. Greene,et al.  How (and where) does moral judgment work? , 2002, Trends in Cognitive Sciences.

[72]  Erkki Sutinen,et al.  Are They Different? Affect, Feeling, Emotion, Sentiment, and Opinion Detection in Text , 2014, IEEE Transactions on Affective Computing.

[73]  Hua Dai,et al.  Explaining consumer satisfaction of services: The role of innovativeness and emotion in an electronic mediated environment , 2015, Decis. Support Syst..

[74]  Lyle H. Ungar,et al.  Modelling Valence and Arousal in Facebook posts , 2016, WASSA@NAACL-HLT.

[75]  Iyad Rahwan,et al.  Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm , 2017, EMNLP.

[76]  Erik Cambria,et al.  EmoSenticSpace: A novel framework for affective common-sense reasoning , 2014, Knowl. Based Syst..

[77]  Diana Inkpen,et al.  Using a Heterogeneous Dataset for Emotion Analysis in Text , 2011, Canadian Conference on AI.

[78]  Suhang Wang,et al.  Fake News Detection on Social Media: A Data Mining Perspective , 2017, SKDD.

[79]  Carlo Strapparava,et al.  WordNet Affect: an Affective Extension of WordNet , 2004, LREC.

[80]  Carlo Strapparava,et al.  SemEval-2007 Task 14: Affective Text , 2007, Fourth International Workshop on Semantic Evaluations (SemEval-2007).

[81]  P. Wilson,et al.  The Nature of Emotions , 2012 .

[82]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[83]  Victoria L. Rubin,et al.  Towards News Verification: Deception Detection Methods for News Discourse , 2015 .

[84]  Saif Mohammad,et al.  SemEval-2018 Task 1: Affect in Tweets , 2018, *SEMEVAL.

[85]  Matthew Leighton Williams,et al.  Cyber Hate Speech on Twitter: An Application of Machine Classification and Statistical Modeling for Policy and Decision Making , 2015 .

[86]  K. Scherer,et al.  How universal and specific is emotional experience? Evidence from 27 countries on five continents , 1986 .

[87]  Andrés Montoyo,et al.  Detecting implicit expressions of emotion in text: A comparative analysis , 2012, Decis. Support Syst..

[88]  Erik Cambria,et al.  A review of affective computing: From unimodal analysis to multimodal fusion , 2017, Inf. Fusion.