How do Humans Interact with Chatbots?: An Analysis of Transcripts

Chatbots enable machines to emulate human conversation, and recent developments have resulted in many online systems for the public to use. Although a few studies have investigated how humans interact with such programs, we are not aware of any that have analyzed transcripts in depth. In this study, students interacted with two Web-based chatbots, Rose and Mitsuku, for five minutes and evaluated how well they thought the software emulated human conversation. We reviewed the transcripts and found that students used fairly simple language and made many text errors. There were no significant differences between the two systems in our experimental measures, but we found that Rose tended to change the topic more often and Mitsuku seemed more argumentative.

[1]  Michael L. Mauldin,et al.  CHATTERBOTS, TINYMUDS, and the Turing Test: Entering the Loebner Prize Competition , 1994, AAAI.

[2]  Morgan C. Benton,et al.  Evaluating Quality of Chatbots and Intelligent Conversational Agents , 2017, ArXiv.

[3]  C. Lortie,et al.  Judgment of the Humanness of an Interlocutor Is in the Eye of the Beholder , 2011, PloS one.

[4]  R. Flesch.  A new readability yardstick. , 1948, The Journal of applied psychology.

[5]  Varol Akman,et al.  Turing Test: 50 Years Later , 2000, Minds and Machines.

[6]  W. R. Ford,et al.  Real conversations with artificial intelligence: A comparison between human-human online conversations and human-chatbot conversations , 2015, Comput. Hum. Behav..

[7]  Yi Mou,et al.  The media inequality: Comparing the initial human-human and human-AI social interactions , 2017, Comput. Hum. Behav..

[8]  Kevin Warwick,et al.  Human misidentification in Turing tests , 2015, J. Exp. Theor. Artif. Intell..

[9]  Wlodek Zadrozny,et al.  Natural language dialogue for personalized interaction , 2000, CACM.

[10]  C. Nass,et al.  Machines and Mindlessness , 2000 .

[11]  Karolina Kuligowska,et al.  Commercial Chatbot: Performance Evaluation, Usability Metrics and Quality Standards of Embodied Conversational Agents , 2015 .

[12]  Eric Atwell,et al.  Chatbots: Are they Really Useful? , 2007, LDV Forum.

[13]  Gina Neff,et al.  Talking to Bots: Symbiotic Agency and the Case of Tay , 2016 .

[14]  Stuart M. Shieber.  On Loebner's lessons , 1994, CACM.

[15]  Avelino J. Gonzalez,et al.  Towards a method for evaluating naturalness in conversational dialog systems , 2009, 2009 IEEE International Conference on Systems, Man and Cybernetics.

[16]  John Zakos,et al.  CLIVE - An Artificially Intelligent Chat Robot for Conversational Language Practice , 2008, SETN.

[17]  Prachi Dwivedi,et al.  User Adaptive Chatbot for Mitigating Depression , 2018 .

[18]  Ong Sing Goh,et al.  A Black-box Approach for Response Quality Evaluation of Conversational Agent Systems , 2007 .

[19]  John P. McIntire,et al.  Methods for chatbot detection in distributed text-based communications , 2010, 2010 International Symposium on Collaborative Technologies and Systems.

[20]  Luke K. Fryer,et al.  Bots as Language Learning Tools , 2006 .

[21]  A. M. Turing,et al.  Computing Machinery and Intelligence , 1950, The Philosophy of Artificial Intelligence.

[22]  Robert Dale,et al.  The return of the chatbots , 2016, Natural Language Engineering.

[23]  Joseph Weizenbaum,et al.  ELIZA—a computer program for the study of natural language communication between man and machine , 1966, CACM.

[24]  Asbjørn Følstad,et al.  Why People Use Chatbots , 2017, INSCI.

[25]  Matteo Turilli,et al.  Turing’s Imitation Game: Still an Impossible Challenge for All Machines and Some Judges––An Evaluation of the 2008 Loebner Contest , 2008, Minds and Machines.

[26]  Antonella De Angeli,et al.  I hate you! Disinhibition with virtual partners , 2008, Interact. Comput..

[27]  Milam W. Aiken,et al.  Flaming in electronic communication , 2004, Decis. Support Syst..

[28]  David M. W. Powers,et al.  The Total Turing Test and the Loebner Prize , 1998, CoNLL.