A Large-scale Analysis of Mixed Initiative in Information-Seeking Dialogues for Conversational Search

Conversational search is a relatively young area of research that aims at automating an information-seeking dialogue. In this article, we help to position it with respect to other research areas within conversational artificial intelligence (AI) by analysing the structural properties of an information-seeking dialogue. To this end, we perform a large-scale dialogue analysis of more than 150K transcripts from 16 publicly available dialogue datasets. These datasets were collected to inform different dialogue-based tasks including conversational search. We extract different patterns of mixed initiative from these dialogue transcripts and use them to compare dialogues of different types. Moreover, we contrast the patterns found in information-seeking dialogues that are being used for research purposes with the patterns found in virtual reference interviews that were conducted by professional librarians. The insights we provide (1) establish close relations between conversational search and other conversational AI tasks and (2) uncover limitations of existing conversational datasets to inform future data collection tasks.

[1]  Nicholas J. Belkin,et al.  Ask for Information Retrieval: Part II. Results of a Design Study , 1982, J. Documentation.

[2]  Joelle Pineau,et al.  Learning an Unreferenced Metric for Online Dialogue Evaluation , 2020, ACL.

[3]  Joelle Pineau,et al.  The Second Conversational Intelligence Challenge (ConvAI2) , 2019, The NeurIPS '18 Competition.

[4]  Seungwhan Moon,et al.  OpenDialKG: Explainable Conversational Reasoning with Attention-based Walks over Knowledge Graphs , 2019, ACL.

[5]  Alfred Kobsa,et al.  User Modeling and User-Adapted Interaction , 1994, User Modeling and User-Adapted Interaction.

[6]  Nicholas J. Belkin,et al.  Cases, scripts, and information-seeking strategies: On the design of interactive information retrieval systems , 1995 .

[7]  Filip Radlinski,et al.  A Theoretical Framework for Conversational Search , 2017, CHIIR.

[8]  Hideo Joho,et al.  Conversational Search (Dagstuhl Seminar 19461) , 2019, Dagstuhl Reports.

[9]  Daniel McDuff,et al.  Theories of Conversation for Conversational IR , 2021, ACM Trans. Inf. Syst..

[10]  Li Chen,et al.  A Survey on Conversational Recommender Systems , 2021, ACM Comput. Surv..

[11]  Joelle Pineau,et al.  The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systems , 2015, SIGDIAL Conference.

[12]  Craig H. Martell,et al.  Lexical and Discourse Analysis of Online Chat Dialog , 2007, International Conference on Semantic Computing (ICSC 2007).

[13]  M. de Rijke,et al.  QRFA: A Data-Driven Model of Information-Seeking Dialogues , 2018, ECIR.

[14]  Lihong Li,et al.  Neural Approaches to Conversational AI , 2019, Found. Trends Inf. Retr..

[15]  Lynn Silipigni Connaway,et al.  Not dead yet! A longitudinal study of query type and ready reference accuracy in live chat and IM reference , 2013 .

[16]  Svitlana Vakulenko,et al.  Knowledge-based Conversational Search , 2019, ArXiv.

[17]  Filip Radlinski,et al.  Coached Conversational Preference Elicitation: A Case Study in Understanding Movie Preferences , 2019, SIGdial.

[18]  Mark Sanderson,et al.  Informing the Design of Spoken Conversational Search: Perspective Paper , 2018, CHIIR.

[19]  Marilyn A. Walker,et al.  Mixed Initiative in Dialogue: An Investigation into Discourse Segmentation , 1990, ACL.

[20]  Jennifer Chu-Carroll,et al.  An Evidential Model for Tracking Initiative in Collaborative Dialogue Interactions , 1998, User Modeling and User-Adapted Interaction.

[21]  Adelheit Stein,et al.  Modelling the Illocutionary Aspects of Information-Seeking Dialogues , 1992, Inf. Process. Manag..

[22]  Daniel McDuff,et al.  MISC: A data set of information-seeking conversations , 2017 .

[23]  Robert S. Taylor.  Question-Negotiation and Information Seeking in Libraries , 1968, Coll. Res. Libr..

[24]  Bowen Hui,et al.  What is Initiative? , 1998, User Modeling and User-Adapted Interaction.

[25]  Nicholas J. Belkin,et al.  Using problem structures for driving human-computer dialogues , 1997, RIAO.

[26]  Steven Euijong Whang,et al.  A Survey on Data Collection for Machine Learning: A Big Data - AI Integration Perspective , 2018, IEEE Transactions on Knowledge and Data Engineering.

[27]  Eunsol Choi,et al.  QuAC: Question Answering in Context , 2018, EMNLP.

[28]  Paul N. Bennett,et al.  Generating Clarifying Questions for Information Retrieval , 2020, WWW.

[29]  Amanda Spink,et al.  Users and Intermediaries in Information Retrieval: What Are They Talking About? , 1997 .

[30]  Danqi Chen,et al.  CoQA: A Conversational Question Answering Challenge , 2018, TACL.

[31]  Omer Levy,et al.  RoBERTa: A Robustly Optimized BERT Pretraining Approach , 2019, ArXiv.

[32]  Claudia Hauff,et al.  Introducing MANtIS: a novel Multi-Domain Information Seeking Dialogues Dataset , 2019, ArXiv.

[33]  Hua Wu,et al.  PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable , 2020, ACL.

[34]  W. Bruce Croft,et al.  Analyzing and Characterizing User Intent in Information-seeking Conversations , 2018, SIGIR.

[35]  Nicholas J. Belkin,et al.  Ask for Information Retrieval: Part I. Background and Theory , 1997, J. Documentation.

[36]  Xiaoyu Shen,et al.  DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset , 2017, IJCNLP.

[37]  Jason Weston,et al.  What makes a good conversation? How controllable attributes affect human judgments , 2019, NAACL.

[38]  Jason Weston,et al.  Wizard of Wikipedia: Knowledge-Powered Conversational agents , 2018, ICLR.

[39]  Jon Atle Gulla,et al.  User-Tailored Planning of Mixed Initiative Information-Seeking Dialogues , 2004, User Modeling and User-Adapted Interaction.

[40]  Quoc V. Le,et al.  Towards a Human-like Open-Domain Chatbot , 2020, ArXiv.

[41]  M. de Rijke,et al.  An Analysis of Mixed Initiative and Collaboration in Information-Seeking Dialogues , 2020, SIGIR.

[42]  Stefan Ultes,et al.  MultiWOZ - A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling , 2018, EMNLP.

[43]  Christopher Joseph Pal,et al.  Towards Deep Conversational Recommendations , 2018, NeurIPS.

[44]  W. Bruce Croft,et al.  Asking Clarifying Questions in Open-Domain Information-Seeking Conversations , 2019, SIGIR.

[45]  Hideo Joho,et al.  Towards a Model for Spoken Conversational Search , 2019, Inf. Process. Manag..

[46]  Martin Halvey,et al.  Conceptualizing agent-human interactions during the conversational search process , 2018 .

[47]  M. de Rijke,et al.  Advances and Challenges in Conversational Recommender Systems: A Survey , 2021, AI Open.