Modeling and Computational Characterization of Twitter Customer Service Conversations

Given the increasing popularity of customer service dialogue on Twitter, analysis of conversation data is essential to understanding trends in customer and agent behavior for the purpose of automating customer service interactions. In this work, we develop a novel taxonomy of fine-grained “dialogue acts” frequently observed in customer service, showcasing acts that are more suited to the domain than the more generic existing taxonomies. Using a sequential SVM-HMM model, we model conversation flow, predicting the dialogue act of a given turn in real time, and showcase this using our “PredDial” portal. We characterize differences between customer and agent behavior in Twitter customer service conversations and investigate the effect of testing our system on different customer service industries. Finally, we use a data-driven approach to predict important conversation outcomes: customer satisfaction, customer frustration, and overall problem resolution. We show that the type and location of certain dialogue acts in a conversation have a significant effect on the probability of desirable and undesirable outcomes and present actionable rules based on our findings. We explore the correlations between different dialogue acts and the outcome of the conversations in detail using an actionable-rule discovery task by leveraging a state-of-the-art sequential rule mining algorithm while modeling a set of conversations as a set of sequences. The patterns and rules we derive can be used as guidelines for outcome-driven automated customer service platforms.

[1]  J. Fleiss.  Measuring nominal scale agreement among many raters. , 1971 .

[2]  Martin L. Puterman,et al.  Markov Decision Processes: Discrete Stochastic Dynamic Programming , 1994 .

[3]  Dongho Kim,et al.  POMDP-based dialogue manager adaptation to extended domains , 2013, SIGDIAL Conference.

[4]  Dilek Z. Hakkani-Tür,et al.  MODEL ADAPTATION FOR DIALOG ACT TAGGING , 2006, 2006 IEEE Spoken Language Technology Workshop.

[5]  David Konopnicki,et al.  Classifying Emotions in Customer Support Dialogues in Social Media , 2016, SIGDIAL Conference.

[6]  Kristy Elizabeth Boyer,et al.  Dialogue Act Modeling in a Complex Task-Oriented Domain , 2010, SIGDIAL Conference.

[7]  Timothy Baldwin,et al.  Classifying Dialogue Acts in One-on-One Live Chats , 2010, EMNLP.

[8]  Kôiti Hasida,et al.  Towards an ISO Standard for Dialogue Act Annotation , 2010, LREC.

[9]  Alan Ritter,et al.  Unsupervised Modeling of Twitter Conversations , 2010, NAACL.

[10]  Tatjana Scheffler,et al.  Dialog Act Annotation for Twitter Conversations , 2015, SIGDIAL Conference.

[11]  Thorsten Joachims,et al.  Cutting-plane training of structural SVMs , 2009, Machine Learning.

[12]  Soroush Vosoughi,et al.  Tweet Acts: A Speech Act Classifier for Twitter , 2016, ICWSM.

[13]  Peter D. Turney,et al.  Emotions Evoked by Common Words and Phrases: Using Mechanical Turk to Create an Emotion Lexicon , 2010, HLT-NAACL 2010.

[14]  David Konopnicki,et al.  Predicting Customer Satisfaction in Customer Support Conversations in Social Media Using Affective Features , 2016, UMAP.

[15]  Andrew McCallum,et al.  Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data , 2001, ICML.

[16]  Edward Ivanovic,et al.  Dialogue Act Tagging for Instant Messaging Chat Sessions , 2005, ACL.

[17]  Susan C. Herring,et al.  Beyond Microblogging: Conversation and Collaboration via Twitter , 2009, 2009 42nd Hawaii International Conference on System Sciences.

[18]  Timothy Baldwin,et al.  Classifying Dialogue Acts in Multi-party Live Chats , 2012, PACLIC.

[19]  Thomas Hofmann,et al.  Hidden Markov Support Vector Machines , 2003, ICML.

[20]  Mark G. Core,et al.  Coding Dialogs with the DAMSL Annotation Scheme , 1997 .

[21]  Mohammed J. Zaki.  Data Mining and Analysis: Fundamental Concepts and Algorithms , 2014 .

[22]  Hans Uszkoreit,et al.  Using Syntactic and Semantic based Relations for Dialogue Act Recognition , 2010, COLING.

[23]  Manuel Campos,et al.  Fast Vertical Mining of Sequential Patterns Using Co-occurrence Information , 2014, PAKDD.

[24]  James F. Allen,et al.  An architecture for more realistic conversational systems , 2001, IUI '01.

[25]  J. R. Landis,et al.  The measurement of observer agreement for categorical data. , 1977, Biometrics.

[26]  Andreas Stolcke,et al.  Dialogue act modeling for automatic tagging and recognition of conversational speech , 2000, CL.

[27]  Amanda Schiffrin,et al.  Modelling speech acts in conversational discourse , 2005 .

[28]  Shafiq R. Joty,et al.  Speech Act Modeling of Written Asynchronous Conversations with Task-Specific Embeddings and Conditional Structured Models , 2016, ACL.

[29]  David DeVault,et al.  Toward incremental dialogue act segmentation in fast-paced interactive dialogue systems , 2016, SIGDIAL Conference.

[30]  Edward Ivanovic,et al.  Using Dialogue Acts to Suggest Responses in Support Services via Instant Messaging , 2006, ALTA.

[31]  Alexander I. Rudnicky,et al.  An Intelligent Assistant for High-Level Task Understanding , 2016, IUI.

[32]  Csr Young,et al.  How to Do Things With Words , 2009 .

[33]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[34]  Ralph Morelli,et al.  A computational speech-act model of human-computer conversations , 1991, Proceedings of the 1991 IEEE Seventeenth Annual Northeast Bioengineering Conference.

[35]  Jalal Mahmud,et al.  "How May I Help You?": Modeling Twitter Customer ServiceConversations Using Fine-Grained Dialogue Acts , 2017, IUI.

[36]  Soroush Vosoughi,et al.  A Semi-Automatic Method for Efficient Detection of Stories on Social Media , 2016, ICWSM.

[37]  Edward Ivanovic,et al.  Automatic instant messaging dialogue using statistical models and dialogue acts , 2008 .

[38]  Wenjie Li,et al.  What Are Tweeters Doing: Recognizing Speech Acts in Twitter , 2011, Analyzing Microtext.

[39]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[40]  Ewan Klein,et al.  Natural Language Processing with Python , 2009 .