An Approach for Ex-Post-Facto Analysis of Knowledge Graph-Driven Chatbots - The DBpedia Chatbot

As chatbots are gaining popularity for simplifying access to information and community interaction, it is essential to examine whether these agents are serving their intended purpose and catering to the needs of their users. Therefore, we present an approach to perform an ex-post-facto analysis over the logs of knowledge base-driven dialogue systems. Using the DBpedia Chatbot as our case study, we inspect three aspects of the interactions, (i) user queries and feedback, (ii) the bot’s response to these queries, and (iii) the overall flow of the conversations. We discuss key implications based on our findings. All the source code used for the analysis can be found at https://github.com/dice-group/DBpedia-Chatlog-Analysis.

[1]  Christopher D. Manning,et al.  Improving Coreference Resolution by Learning Entity-Level Distributed Representations , 2016, ACL.

[2]  Robert Dale,et al.  The return of the chatbots , 2016, Natural Language Engineering.

[3]  Kuldeep Singh,et al.  Qanary - A Methodology for Vocabulary-Driven Open Question Answering Systems , 2016, ESWC.

[4]  Stephen J. Cox,et al.  Analysis of User Interaction with Service Oriented Chatbot Systems , 2007, HCI.

[5]  Ray Kurzweil,et al.  Multilingual Universal Sentence Encoder for Semantic Retrieval , 2019, ACL.

[6]  Ida Maria Haugstveit,et al.  Help! Is my chatbot falling into the uncanny valley? An empirical study of user experience in human-chatbot interaction , 2019, Human Technology.

[7]  Gerhard Weikum,et al.  ComQA: A Community-sourced Dataset for Complex Factoid Question Answering with Paraphrase Clusters , 2018, NAACL.

[8]  Zahra Ashktorab,et al.  Resilient Chatbots: Repair Strategy Preferences for Conversational Breakdowns , 2019, CHI.

[9]  Axel-Cyrille Ngonga Ngomo,et al.  Enhancing Community Interactions with Data-Driven Chatbots--The DBpedia Chatbot , 2018, WWW.

[10]  Timothy Dozat,et al.  Universal Dependency Parsing from Scratch , 2019, CoNLL.

[11]  Jens Lehmann,et al.  Introduction to Neural Network based Approaches for Question Answering over Knowledge Graphs , 2019, ArXiv.

[12]  Tomas Mikolov,et al.  Advances in Pre-Training Distributed Word Representations , 2017, LREC.

[13]  Geoffrey E. Hinton,et al.  Visualizing Data using t-SNE , 2008 .

[14]  Shwetak N. Patel,et al.  Evaluating and Informing the Design of Chatbots , 2018, Conference on Designing Interactive Systems.

[15]  Julia Neidhardt,et al.  The Potential of Chatbots: Analysis of Chatbot Conversations , 2019, 2019 IEEE 21st Conference on Business Informatics (CBI).

[16]  Vadim Savenkov,et al.  Open Data Chatbot , 2019, ESWC.

[17]  Leland McInnes,et al.  hdbscan: Hierarchical density based clustering , 2017, J. Open Source Softw..

[18]  Christopher D. Manning,et al.  Deep Reinforcement Learning for Mention-Ranking Coreference Models , 2016, EMNLP.

[19]  Yung-Ju Chang,et al.  When There is No Progress with a Task-Oriented Chatbot: A Conversation Analysis , 2019, MobileHCI.

[20]  Pierre Maret,et al.  QAnswer: A Question Answering prototype bridging the gap between a considerable part of the LOD cloud and end-users , 2019, WWW.

[21]  Jens Lehmann,et al.  DBpedia - A large-scale, multilingual knowledge base extracted from Wikipedia , 2015, Semantic Web.

[22]  Andreas Lommatzsch,et al.  An Information Retrieval-based Approach for Building Intuitive Chatbots for Large Knowledge Bases , 2019, LWDA.

[23]  Adriano Rivolli,et al.  KnowBots: Discovering Relevant Patterns in Chatbot Dialogues , 2019, DS.

[24]  Pablo N. Mendes,et al.  Improving efficiency and accuracy in multilingual entity extraction , 2013, I-SEMANTICS '13.

[25]  M. de Rijke,et al.  Message Passing for Complex Question Answering over Knowledge Graphs , 2019, CIKM.

[26]  Ming Zhou,et al.  Extracting Chatbot Knowledge from Online Discussion Forums , 2007, IJCAI.