A Conversational Agent to Improve Response Quality in Course Evaluations

Recent advances in Natural Language Processing (NLP) bear the opportunity to design new forms of human-computer interaction with conversational interfaces. We hypothesize that these interfaces can interactively engage students to increase response quality of course evaluations in education compared to the common standard of web surveys. Past research indicates that web surveys come with disadvantages, such as poor response quality caused by inattention, survey fatigue or satisficing behavior. To test if conversational interfaces have a positive impact on the level of enjoyment and the response quality, we design an NLP-based conversational agent and deploy it in a field experiment with 127 students in our lecture and compare it with a web survey as a baseline. Our findings indicate that using conversational agents for evaluations are resulting in higher levels of response quality and level of enjoyment, and are therefore, a promising approach to increase the effectiveness of surveys in general.

[1]  Matthias Söllner,et al.  Unleashing the Potential of Chatbots in Education: A State-Of-The-Art Analysis , 2018, Academy of Management Proceedings.

[2]  C. O. Houle.  : Marks of Readable Style: A Study in Adult Education , 1945 .

[3]  Aliane Loureiro Krassmann,et al.  Conversational Agents in Distance Education: Comparing Mood States with Students’ Perception , 2018 .

[4]  Sandra C. Jones,et al.  Online student evaluation improves Course Experience Questionnaire results in a physiotherapy program , 2008 .

[5]  Navneet Kaur,et al.  Opinion mining and sentiment analysis , 2016, 2016 3rd International Conference on Computing for Sustainable Global Development (INDIACom).

[6]  Victoria L. Rubin,et al.  Artificially intelligent conversational agents in libraries , 2010, Libr. Hi Tech.

[7]  Susan Bull,et al.  Bringing Chatbots into education: Towards Natural Language Negotiation of Open Learner Models , 2006, SGAI Conf..

[8]  Justin Scott Giboney,et al.  Facilitating Natural Conversational Agent Interactions: Lessons from a Deception Experiment , 2014, ICIS.

[9]  Youngme Moon.  Intimate Exchanges: Using Computers to Elicit Self-Disclosure from Consumers , 2000 .

[10]  Pushpak Bhattacharyya,et al.  Measuring Sentiment Annotation Complexity of Text , 2014, ACL.

[11]  Thiemo Wambsganss,et al.  Mining User-Generated Repair Instructions from Automotive Web Communities , 2019, HICSS.

[12]  C. Maier,et al.  Chatbot Acceptance in Healthcare: Explaining User Adoption of Conversational Agents for disease Diagnosis , 2019, ECIS.

[13]  Fang-Wu Tung,et al.  Designing social presence in e-learning environments: Testing the effect of interactivity on children , 2006, Interact. Learn. Environ..

[14]  Joonhwan Lee,et al.  Comparing Data from Chatbot and Web Surveys: Effects of Platform and Conversational Style on Survey Response Quality , 2019, CHI.

[15]  Joseph B. Walther,et al.  Selective self-presentation in computer-mediated communication: Hyperpersonal dimensions of technology, language, and cognition , 2007, Comput. Hum. Behav..

[16]  Ewan Klein,et al.  Natural Language Processing with Python , 2009 .

[17]  Alexander Maedche,et al.  Faster is Not Always Better: Understanding the Effect of Dynamic Response Delays in Human-Chatbot Interaction , 2018, ECIS.

[18]  T. Kowatsch,et al.  Text-based Healthcare Chatbots Supporting Patient and Health Professional Teams: Preliminary Results of a Randomized Controlled Trial on Childhood Obesity , 2017 .

[19]  Broderick Crawford,et al.  Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) , 2007 .

[20]  Fang Chen,et al.  Using language complexity to measure cognitive load for adaptive interaction design , 2010, IUI '10.

[21]  C. Nass,et al.  Machines and Mindlessness , 2000 .

[22]  Julia Fink,et al.  Anthropomorphism and Human Likeness in the Design of Robots and Human-Robot Interaction , 2012, ICSR.

[23]  Anbang Xu,et al.  A New Chatbot for Customer Service on Social Media , 2017, CHI.

[24]  Eric Atwell,et al.  Using corpora in machine-learning chatbot systems , 2005 .

[25]  Clifford Nass,et al.  Computers are social actors , 1994, CHI '94.

[26]  Clint Davies,et al.  Eliciting student feedback for course development: the application of a qualitative course evaluation tool among business research students , 2019 .

[27]  Malgorzata Erikson,et al.  Student responses to a reflexive course evaluation , 2016 .

[28]  Thiemo Wambsganss,et al.  Designing a Conversational Agent as a Formative Course Evaluation Tool , 2020, Wirtschaftsinformatik.

[29]  Donggil Song,et al.  Interacting with a conversational agent system for educational purposes in online courses , 2017, 2017 10th International Conference on Human System Interactions (HSI).

[30]  Raphael Meyer von Wolff,et al.  Say Hello to Your New Automated Tutor - A Structured Literature Review on Pedagogical Conversational Agents , 2019, Wirtschaftsinformatik.

[31]  Geert Loosveldt,et al.  Face-to-Face versus Web Surveying in a High-Internet-Coverage Population Differences in Response Quality , 2008 .