Engagement and Usability of Conversational Search - A Study of a Medical Resource Center Chatbot

Due to advances in natural language understanding, chatbots have become popular for assisting users in various tasks, for example, searching. Chatbots allow natural-language queries, which can be useful in case of complex information needs, and they provide a higher level of interactivity by displaying information in a dialog-like format. However, chatbots are often only used as auxiliaries for a graphical search user interface. Thus, they must be engaging and usable so that users both want to and able to use them. In this study, a chatbotbased and a website-based search interface were compared in terms of engagement and usability. Engagement was measured using the User Engagement Scale; think-aloud protocol and a questionnaire were used to assess usability. Behavioral measures were used to triangulate data. Findings indicate that the usage of the chatbot did not lead to a higher level of engagement, moreover, its usability was lower compared to the website-based search interface.

[1]  Gary Marchionini,et al.  Exploratory search , 2006, Commun. ACM.

[2]  Marcia J. Bates,et al.  Information search tactics , 1979, J. Am. Soc. Inf. Sci..

[3]  Jaime Arguello,et al.  Development and Evaluation of Search Tasks for IIR Experiments using a Cognitive Complexity Framework , 2015, ICTIR.

[4]  Thomas Beckers.  Supporting polyrepresentation and information seeking strategies , 2009 .

[5]  Dilek Z. Hakkani-Tür,et al.  Spoken language understanding , 2008, IEEE Signal Processing Magazine.

[6]  Justine Cassell,et al.  Relational agents: a model and implementation of building user trust , 2001, CHI.

[7]  J. Halamka,et al.  Chatbots and Conversational Agents in Mental Health: A Review of the Psychiatric Landscape , 2019, Canadian journal of psychiatry. Revue canadienne de psychiatrie.

[8]  Diane Kelly,et al.  Engaged or Frustrated?: Disambiguating Emotional State in Search , 2017, SIGIR.

[9]  Joseph Weizenbaum,et al.  ELIZA—a computer program for the study of natural language communication between man and machine , 1966, CACM.

[10]  Jack Meadows.  S.C. Bradford and documentation: A review article , 2002 .

[11]  V. Braun,et al.  Using thematic analysis in psychology , 2006 .

[12]  Jeff Sauro,et al.  Average task times in usability tests: what to report? , 2010, CHI 2010.

[13]  Ben Shneiderman,et al.  The eyes have it: a task by data type taxonomy for information visualizations , 1996, Proceedings 1996 IEEE Symposium on Visual Languages.

[14]  Juanan Pereira,et al.  Leveraging chatbots to improve self-guided learning through conversational quizzes , 2016, TEEM.

[15]  Andrew Thompson,et al.  Stimulating task interest: human partners or chatbots? , 2018, Future-proof CALL: language learning as exploration and encounters – short papers from EUROCALL 2018.

[16]  Keeheon Lee,et al.  Can Chatbots Help Reduce the Workload of Administrative Officers? - Implementing and Deploying FAQ Chatbot Service in a University , 2019, HCI.

[17]  J. Chalmers,et al.  Introduction and scope , 2008 .

[18]  Charles L. A. Clarke,et al.  Exploring Conversational Search With Humans, Assistants, and Wizards , 2017, CHI Extended Abstracts.

[19]  A. Colman,et al.  Optimal number of response categories in rating scales: reliability, validity, discriminating power, and respondent preferences. , 2000, Acta psychologica.

[20]  Hai Zhao,et al.  Lingke: a Fine-grained Multi-turn Chatbot for Customer Service , 2018, COLING.

[21]  Mihaly Csikszentmihalyi,et al.  A Theoretical Model for Enjoyment , 2014 .

[22]  Robert G. Capra,et al.  An empirical study of interest, task complexity, and search behaviour on user engagement , 2020, Inf. Process. Manag..

[23]  Enrico Coiera,et al.  The Personalization of Conversational Agents in Health Care: Systematic Review , 2019, Journal of medical Internet research.

[24]  Tony Russell-Rose,et al.  Formulating the Query , 2013 .

[25]  Caroline Bassett,et al.  The computational therapeutic: exploring Weizenbaum’s ELIZA as a history of the present , 2018, AI & SOCIETY.

[26]  Elaine Toms,et al.  What is user engagement? A conceptual framework for defining user engagement with technology , 2008, J. Assoc. Inf. Sci. Technol..

[27]  Raphaël Troncy,et al.  A Survey of Definitions and Models of Exploratory Search , 2017, ESIDA@IUI.

[28]  G. David Garson,et al.  Research Designs , 2011, International Encyclopedia of Statistical Science.

[29]  Gabriele Meiselwitz,et al.  Teacher Agents: The Current State, Future Trends, and Many Roles of Intelligent Agents in Education , 2011, HCI.

[30]  M. de Rijke,et al.  Conversational Exploratory Search via Interactive Storytelling , 2017, ArXiv.

[31]  Rob Procter,et al.  User engagement by user-centred design in e-Health , 2010, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[32]  Angela M. Cirucci,et al.  Usability Testing , 2021, UX Research Methods for Media and Communication Studies.

[33]  Olga Perski,et al.  Does the addition of a supportive chatbot promote user engagement with a smoking cessation app? An experimental study , 2019, Digital health.

[34]  Ryen W. White.  Opportunities and challenges in search interaction , 2018, Commun. ACM.

[35]  R. Lederman,et al.  Artificial Intelligence-Assisted Online Social Therapy for Youth Mental Health , 2017, Front. Psychol..

[36]  Daniel McDuff,et al.  MISC: A data set of information-seeking conversations , 2017 .

[37]  O'BrienHeather,et al.  An empirical evaluation of the User Engagement Scale (UES) in online news environments , 2015 .

[38]  Diane Kelly,et al.  Engagement in Information Search , 2016, Why Engagement Matters.

[39]  Peter Ingwersen,et al.  Cognitive Perspectives of Information Retrieval Interaction: Elements of a Cognitive IR Theory , 1996, J. Documentation.

[40]  H. Lowe,et al.  Understanding and using the medical subject headings (MeSH) vocabulary to perform literature searches. , 1994, JAMA.

[41]  David Traum,et al.  The Information State Approach to Dialogue Management , 2003 .

[42]  Elaine Toms,et al.  Is there a universal instrument for measuring interactive information retrieval?: the case of the user engagement scale , 2010, IIiX.

[43]  Abigail Sellen,et al.  "Like Having a Really Bad PA": The Gulf between User Expectation and Experience of Conversational Agents , 2016, CHI.

[44]  Jimmy J. Lin,et al.  Desiderata for exploratory search interfaces to Web archives in support of scholarly activities , 2016, 2016 IEEE/ACM Joint Conference on Digital Libraries (JCDL).

[45]  Luke K. Fryer,et al.  Stimulating and sustaining interest in a language course: An experimental comparison of Chatbot and Human task partners , 2017, Comput. Hum. Behav..

[46]  KellyDiane,et al.  Questionnaire mode effects in interactive information retrieval experiments , 2008 .

[47]  David Ellis,et al.  A Behavioural Approach to Information Retrieval System Design , 1989, J. Documentation.

[48]  Hilde A. M. Voorveld,et al.  Privacy Concerns in Chatbot Interactions , 2019, CONVERSATIONS.

[49]  Mateusz Dubiel.  Towards Human-Like Conversational Search Systems , 2018, CHIIR.

[50]  James H. Martin,et al.  Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition , 2000 .

[51]  Jakob Nielsen,et al.  Measuring usability: preference vs. performance , 1994, CACM.

[52]  Terry Winograd,et al.  Understanding natural language , 1974 .

[53]  Mingming Zhou,et al.  Gender difference in web search perceptions and behavior: Does it vary by task performance? , 2014, Comput. Educ..

[54]  Robert Dale,et al.  The return of the chatbots , 2016, Natural Language Engineering.

[55]  Cassidy R. Sugimoto,et al.  A systematic review of interactive information retrieval evaluation studies, 1967-2006 , 2013, J. Assoc. Inf. Sci. Technol..

[56]  John Flach.  A Framework for Ecological Interface Design (EID) , 2011 .

[57]  Jeff Sauro,et al.  Correlations among prototypical usability metrics: evidence for the construct of usability , 2009, CHI.

[58]  Menno D. T. de Jong,et al.  Retrospective vs. concurrent think-aloud protocols: Testing the usability of an online library catalogue , 2003, Behav. Inf. Technol..

[59]  K. Fitzpatrick,et al.  Delivering Cognitive Behavior Therapy to Young Adults With Symptoms of Depression and Anxiety Using a Fully Automated Conversational Agent (Woebot): A Randomized Controlled Trial , 2017, JMIR mental health.

[60]  Chauncey E. Wilson.  Questionnaires and Surveys , 2013 .

[61]  Martin Halvey,et al.  Investigating how conversational search agents affect user's behaviour, performance and search experience , 2018 .

[62]  M. Jong,et al.  Conversational Commerce : the Conversation of Tomorrow , 2018 .

[63]  M. Chi,et al.  Gender Differences in Patterns of Searching the Web , 2003 .

[64]  Filip Radlinski,et al.  A Theoretical Framework for Conversational Search , 2017, CHIIR.

[65]  Percy Liang.  Talking to computers in natural language , 2014, XRDS.

[66]  Jonathan Earthy,et al.  New ISO Standards for Usability, Usability Reports and Usability Measures , 2016, HCI.

[67]  Peter Thomas,et al.  Chapter 3 – The Very Idea: Informing HCI Design from Conversation Analysis , 1990 .

[68]  Eunju Ko,et al.  Chatbot e-service and customer satisfaction regarding luxury brands , 2020 .

[69]  Iris Xie.  Interactive IR in OPAC Environments , 2008 .

[70]  Sandra Lutz Hochreutener,et al.  Self-Anamnesis with a Conversational User Interface: Concept and Usability Study , 2018, Methods of Information in Medicine.

[71]  Irene Celino,et al.  Submitting surveys via a conversational interface: an evaluation of user acceptance and approach effectiveness , 2020, Int. J. Hum. Comput. Stud..

[72]  Trent W. Lewis,et al.  Designing and Evaluating Interactive Agents as Social Skills Tutors for Children with Autism Spectrum Disorder , 2011 .

[73]  Peter Ingwersen,et al.  Polyrepresentation of information needs and semantic entities: elements of a cognitive theory for information retrieval interaction , 1994, SIGIR '94.

[74]  Kasper Hornbæk,et al.  Current practice in measuring usability: Challenges to usability studies and research , 2006, Int. J. Hum. Comput. Stud..

[75]  Iris Xie.  User-Oriented IR Research Approaches , 2008 .

[76]  S. Shyam Sundar,et al.  Theoretical Importance of Contingency in Human-Computer Interaction , 2016, Commun. Res..

[77]  Shyam Sundar,et al.  Social psychology of interactivity in human-website interaction , 2009 .

[78]  Ioannis Arapakis,et al.  Theories, methods and current research on emotions in library and information science, information retrieval and human-computer interaction , 2011, Inf. Process. Manag..

[79]  Ted Boren,et al.  Thinking aloud: reconciling theory and practice , 2000 .

[80]  Peter Ingwersen,et al.  The development of a method for the evaluation of interactive information retrieval systems , 1997, J. Documentation.

[81]  Heather L. O'Brien,et al.  Theoretical Perspectives on User Engagement , 2016, Why Engagement Matters.

[82]  D. Gratzer,et al.  Open for Business: Chatbots, E-therapies, and the Future of Psychiatry , 2019, Canadian journal of psychiatry. Revue canadienne de psychiatrie.

[83]  Justine Cassell,et al.  Embodied conversational interface agents , 2000, CACM.

[84]  G. A. Miller.  THE PSYCHOLOGICAL REVIEW THE MAGICAL NUMBER SEVEN, PLUS OR MINUS TWO: SOME LIMITS ON OUR CAPACITY FOR PROCESSING INFORMATION 1 , 1956 .

[85]  Eric N. Wiebe,et al.  Measuring engagement in video game-based environments: Investigation of the User Engagement Scale , 2014, Comput. Hum. Behav..

[86]  Wanda Pratt,et al.  Transparent Queries: investigation users' mental models of search engines , 2001, SIGIR '01.

[87]  Dror Ben-Zeev,et al.  Analyzing mHealth Engagement: Joint Models for Intensively Collected User Engagement Data , 2017, JMIR mHealth and uHealth.

[88]  Nicholas J. Belkin,et al.  On the evaluation of interactive information retrieval systems , 2010 .

[89]  Tony Russell-Rose,et al.  Displaying and Manipulating Results , 2013 .

[90]  Marcia J. Bates,et al.  The design of browsing and berrypicking techniques for the online search interface , 1989 .

[91]  B. Weijters,et al.  The effect of rating scale format on response styles: the number of response categories and response catgory labels , 2010 .

[92]  Kien Hoa Ly,et al.  A fully automated conversational agent for promoting mental well-being: A pilot RCT using mixed methods , 2017, Internet interventions.

[93]  David Henry Ward.  Why Users Choose Chat , 2005 .

[94]  Kaori Nakao,et al.  Chatbot learning partners: Connecting learning experiences, interest and competence , 2019, Comput. Hum. Behav..

[95]  David Griol,et al.  Conversational Interfaces: Past and Present , 2016 .

[96]  Arthur C. Graesser,et al.  Learning by Communicating in Natural Language With Conversational Agents , 2014 .

[97]  Paul A. Cairns,et al.  A practical approach to measuring user engagement with the refined user engagement scale (UES) and new UES short form , 2018, Int. J. Hum. Comput. Stud..

[98]  Alexander I. Rudnicky,et al.  A principled approach for rejection threshold optimization in spoken dialog systems , 2005, INTERSPEECH.

[99]  Jack Minker,et al.  Information storage and retrieval: a survey and functional description , 1977, SIGF.

[100]  Diane Kelly,et al.  Methods for Evaluating Interactive Information Retrieval Systems with Users , 2009, Found. Trends Inf. Retr..

[101]  David G. Novick,et al.  Natural-language interfaces , 2000, CHI Extended Abstracts.

[102]  Sándor Dominich.  Basics of Information Retrieval Technology , 2008 .

[103]  Elaine Toms,et al.  The development and evaluation of a survey to measure user engagement , 2010, J. Assoc. Inf. Sci. Technol..

[104]  Alistair Sutcliffe,et al.  Designing for User Experience and Engagement , 2016, Why Engagement Matters.

[105]  DeeAnn Allison,et al.  Chatbots in the Library: is it time? , 2012, Libr. Hi Tech.

[106]  Jakob Nielsen,et al.  Usability , 2009 .

[107]  N. Baumann.  How to use the medical subject headings (MeSH) , 2016, International journal of clinical practice.

[108]  Kasper Hornbæk,et al.  Meta-analysis of correlations among usability measures , 2007, CHI.

[109]  Pia Borlund,et al.  The IIR evaluation model: a framework for evaluation of interactive information retrieval systems , 2003, Inf. Res..

[110]  Daniel McDuff,et al.  Style and Alignment in Information-Seeking Conversation , 2018, CHIIR.

[111]  William A. Woods,et al.  Progress in natural language understanding: an application to lunar geology , 1973, AFIPS National Computer Conference.

[112]  Russell Fulmer,et al.  Using Psychological Artificial Intelligence (Tess) to Relieve Symptoms of Depression and Anxiety: Randomized Controlled Trial , 2018, JMIR mental health.