QRFA: A Data-Driven Model of Information-Seeking Dialogues

Understanding the structure of interaction processes helps us to improve information-seeking dialogue systems. Analyzing an interaction process boils down to discovering patterns in sequences of alternating utterances exchanged between a user and an agent. Process mining techniques have been successfully applied to analyze structured event logs, discovering the underlying process models or evaluating whether the observed behavior is in conformance with the known process. In this paper, we apply process mining techniques to discover patterns in conversational transcripts and extract a new model of information-seeking dialogues, QRFA, for Query, Request, Feedback, Answer. Our results are grounded in an empirical evaluation across multiple conversational datasets from different domains, which was never attempted before. We show that the QRFA model better reflects conversation flows observed in real information-seeking conversations than models proposed previously. Moreover, QRFA allows us to identify malfunctioning in dialogue system transcripts as deviations from the expected conversation flow described by the model via conformance analysis.

[1]  Filip Radlinski,et al.  A Theoretical Framework for Conversational Search , 2017, CHIIR.

[2]  E. Schegloff.  Sequencing in Conversational Openings , 1968 .

[3]  M. de Rijke,et al.  Measuring Semantic Coherence of a Conversation , 2018, SEMWEB.

[4]  Massimo Mecella,et al.  A two-step fast algorithm for the automated discovery of declarative workflows , 2013, 2013 IEEE Symposium on Computational Intelligence and Data Mining (CIDM).

[5]  Terry Winograd,et al.  Understanding computers and cognition - a new foundation for design , 1987 .

[6]  Nicholas J. Belkin,et al.  Cases, scripts, and information-seeking strategies: On the design of interactive information retrieval systems , 1995 .

[7]  Flávia Maria Santoro,et al.  Analysis of Knowledge-Intensive Processes Focused on the Communication Perspective , 2017, BPM.

[8]  Carolyn Penstein Rosé,et al.  Modeling Dialogue Acts with Content Word Filtering and Speaker Preferences , 2017, EMNLP.

[9]  Sander J. J. Leemans,et al.  Process and Deviation Exploration with Inductive Visual Miner , 2014, BPM.

[10]  Csr Young,et al.  How to Do Things With Words , 2009 .

[11]  Mark Sanderson,et al.  How Do People Interact in Conversational Speech-Only Search Tasks: A Preliminary Analysis , 2017, CHIIR.

[12]  M. de Rijke,et al.  Dialogue Generation: From Imitation Learning to Inverse Reinforcement Learning , 2018, AAAI.

[13]  Adelheit Stein,et al.  Modelling the Illocutionary Aspects of Information-Seeking Dialogues , 1992, Inf. Process. Manag..

[14]  Gary Geunbae Lee,et al.  Semi-supervised Speech Act Recognition in Emails and Forums , 2009, EMNLP.

[15]  Massimo Mecella,et al.  Mining Artful Processes from Knowledge Workers' Emails , 2013, IEEE Internet Computing.

[16]  John R. Searle,et al.  Speech Acts: An Essay in the Philosophy of Language , 1970 .

[17]  Mark Sanderson,et al.  Informing the Design of Spoken Conversational Search: Perspective Paper , 2018, CHIIR.

[18]  Weiguo Fan,et al.  An Analytical Framework for Understanding Knowledge-Sharing Processes in Online Q&A Communities , 2014, ACM Trans. Manag. Inf. Syst..

[19]  M. Christopher Riley.  Anarchy, State, or Utopia?: Checks and Balances in Internet Governance , 2013, IEEE Internet Computing.

[20]  Mark G. Core,et al.  Coding Dialogs with the DAMSL Annotation Scheme , 1997 .

[21]  Wil M.P. van der Aalst,et al.  Discovery of Frequent Episodes in Event Logs , 2014, SIMPDA.

[22]  Antoine Raux,et al.  The Dialog State Tracking Challenge Series , 2014, AI Mag..

[23]  Boudewijn F. van Dongen,et al.  Conformance Checking Using Cost-Based Fitness Analysis , 2011, 2011 IEEE 15th International Enterprise Distributed Object Computing Conference.

[24]  Tsuyoshi Murata,et al.  {m , 1934, ACML.

[25]  Andreas Stolcke,et al.  Dialogue act modeling for automatic tagging and recognition of conversational speech , 2000, CL.

[26]  Wil M. P. van der Aalst,et al.  The Application of Petri Nets to Workflow Management , 1998, J. Circuits Syst. Comput..

[27]  Wil M. P. van der Aalst,et al.  Process Mining , 2016, Springer Berlin Heidelberg.

[28]  Tom M. Mitchell,et al.  Learning to Classify Email into “Speech Acts” , 2004, EMNLP.

[29]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[30]  Massimo Mecella,et al.  On the Discovery of Declarative Control Flows for Artful Processes , 2015, ACM Trans. Manag. Inf. Syst..

[31]  A. J. Wootton,et al.  Erving Goffman: Exploring the interaction order. , 1992 .

[32]  Boudewijn F. van Dongen,et al.  Replaying history on process models for conformance checking and performance analysis , 2012, WIREs Data Mining Knowl. Discov..

[33]  D. Schiffrin.  Approaches to Discourse , 1997 .

[34]  J. Searle.  A classification of illocutionary acts , 1976, Language in Society.

[35]  Antoine Raux,et al.  The Dialog State Tracking Challenge Series: A Review , 2016, Dialogue Discourse.

[36]  R. Rosenfeld,et al.  Cases , 2010, Otolaryngology--head and neck surgery : official journal of American Academy of Otolaryngology-Head and Neck Surgery.