Unleashing the Potential of Conversational Agents for Course Evaluations: Empirical Insights from a Comparison with Web Surveys

Recent advances in Natural Language Processing (NLP) bear the opportunity to design new forms of human-computer interaction with conversational interfaces. However, little is known about how these interfaces change the way users respond in online course evaluations. We aim to explore the effects of conversational agents (CAs) on the response quality of online course evaluations in education compared to the common standard of web surveys. Past research indicates that web surveys come with disadvantages, such as poor response quality caused by inattention, survey fatigue or satisficing behavior. We propose that a conversational interface will have a positive effect on the response quality through the different way of interaction. To test our hypotheses, we design an NLP-based CA and deploy it in a field experiment with 176 students in three different course formats and compare it with a web survey as a baseline. The results indicate that participants using the CA showed higher levels of response quality and social presence compared to the web survey. These findings along with technology acceptance measurements suggest that using CAs for evaluation are a promising approach to increase the effectiveness of surveys in general.

[1]  Raphael Meyer von Wolff,et al.  Say Hello to Your New Automated Tutor - A Structured Literature Review on Pedagogical Conversational Agents , 2019, Wirtschaftsinformatik.

[2]  Malgorzata Erikson,et al.  Student responses to a reflexive course evaluation , 2016 .

[3]  Thiemo Wambsganss,et al.  Designing a Conversational Agent as a Formative Course Evaluation Tool , 2020, Wirtschaftsinformatik.

[4]  Julia Fink,et al.  Anthropomorphism and Human Likeness in the Design of Robots and Human-Robot Interaction , 2012, ICSR.

[5]  Anbang Xu,et al.  A New Chatbot for Customer Service on Social Media , 2017, CHI.

[6]  J. Cacioppo,et al.  On seeing human: a three-factor theory of anthropomorphism. , 2007, Psychological review.

[7]  Geert Loosveldt,et al.  Face-to-Face versus Web Surveying in a High-Internet-Coverage Population Differences in Response Quality , 2008 .

[8]  Louis-Philippe Morency,et al.  It's only a computer: Virtual humans increase willingness to disclose , 2014, Comput. Hum. Behav..

[9]  Jan Marco Leimeister,et al.  Future Work and Enterprise Systems , 2018, Bus. Inf. Syst. Eng..

[10]  Viswanath Venkatesh,et al.  Technology Acceptance Model 3 and a Research Agenda on Interventions , 2008, Decis. Sci..

[11]  Sandra C. Jones,et al.  Online student evaluation improves Course Experience Questionnaire results in a physiotherapy program , 2008 .

[12]  Edward M. Latorre-Navarro.  An Intelligent Natural Language Conversational System for Academic Advising , 2015 .

[13]  T. Kowatsch,et al.  Text-based Healthcare Chatbots Supporting Patient and Health Professional Teams: Preliminary Results of a Randomized Controlled Trial on Childhood Obesity , 2017 .

[14]  Karen Swan,et al.  ON THE NATURE AND DEVELOPMENT OF SOCIAL PRESENCE IN ONLINE COURSE DISCUSSIONS , 2019, Online Learning.

[15]  ChanMin Kim,et al.  A Virtual Change Agent: Motivating Pre-service Teachers to Integrate Technology in Their Future Classrooms , 2008, J. Educ. Technol. Soc..

[16]  Sabine Payr.  The virtual university's faculty: An overview of educational agents , 2003, Appl. Artif. Intell..

[17]  Liane Margarida Rockenbach Tarouco,et al.  Supporting Problem-Solving in Mathematics with a Conversational Agent Capable of Representing Gifted Students' Knowledge , 2014, 2014 47th Hawaii International Conference on System Sciences.

[18]  Erik Blair,et al.  Improving higher education practice through student evaluation systems: is the student voice being heard? , 2014 .

[19]  Broderick Crawford,et al.  Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) , 2007 .

[20]  Theo Araujo,et al.  Living up to the chatbot hype: The influence of anthropomorphic design cues and communicative agency framing on conversational agent and company perceptions , 2018, Comput. Hum. Behav..

[21]  Eric Atwell,et al.  Using corpora in machine-learning chatbot systems , 2005 .

[22]  Ewan Klein,et al.  Natural Language Processing with Python , 2009 .

[23]  Ann Colley,et al.  Style and Content in E-Mails and Letters to Male and Female Friends , 2004 .

[24]  Jiebo Luo,et al.  Touch Your Heart: A Tone-aware Chatbot for Customer Care on Social Media , 2018, CHI.

[25]  C. Maier,et al.  Chatbot Acceptance in Healthcare: Explaining User Adoption of Conversational Agents for disease Diagnosis , 2019, ECIS.

[26]  J. Reid.  Computer-assisted instruction. , 1993, Missouri medicine.

[27]  Fang Chen,et al.  Using language complexity to measure cognitive load for adaptive interaction design , 2010, IUI '10.

[28]  C. Nass,et al.  Machines and Mindlessness , 2000 .

[29]  Justin Scott Giboney,et al.  Facilitating Natural Conversational Agent Interactions: Lessons from a Deception Experiment , 2014, ICIS.

[30]  Richard C. Atkinson,et al.  Human Memory: A Proposed System and its Control Processes , 1968, Psychology of Learning and Motivation.

[31]  Frank Biocca,et al.  The Effect of the Agency and Anthropomorphism on Users' Sense of Telepresence, Copresence, and Social Presence in Virtual Environments , 2003, Presence: Teleoperators & Virtual Environments.

[32]  Thrasyvoulos Tsiatsos,et al.  A Configurable Conversational Agent to Trigger Students’ Productive Dialogue: A Pilot Study in the CALL Domain , 2013, International Journal of Artificial Intelligence in Education.

[33]  Aliane Loureiro Krassmann,et al.  Conversational Agents in Distance Education: Comparing Mood States with Students’ Perception , 2018 .

[34]  Shirley Williams,et al.  Design and Implementation of Conversational Agents for Harvesting Feedback in eLearning Systems , 2013, EC-TEL.

[35]  K. Fitzpatrick,et al.  Delivering Cognitive Behavior Therapy to Young Adults With Symptoms of Depression and Anxiety Using a Fully Automated Conversational Agent (Woebot): A Randomized Controlled Trial , 2017, JMIR mental health.

[36]  Gordon B. Davis,et al.  User Acceptance of Information Technology: Toward a Unified View , 2003, MIS Q..

[37]  Jay F. Nunamaker,et al.  The influence of conversational agent embodiment and conversational relevance on socially desirable responding , 2018, Decis. Support Syst..

[38]  Thiemo Wambsganss,et al.  Towards a Taxonomy of Text Mining Features , 2019, ECIS.

[39]  Tibert Verhagen,et al.  Virtual Customer Service Agents: Using Social Presence and Personalization to Shape Online Service Encounters , 2014, J. Comput. Mediat. Commun..

[40]  J. Krosnick.  Response strategies for coping with the cognitive demands of attitude measures in surveys , 1991 .

[41]  Jan Marco Leimeister,et al.  Alexa, Can You Help Us Solve This Problem?: How Conversations With Smart Personal Assistant Tutors Increase Task Group Outcomes , 2019, CHI Extended Abstracts.

[42]  Chenicheri Sid Nair,et al.  Benchmarking for the effective use of student evaluation data , 2015 .

[43]  Donggil Song,et al.  Interacting with a conversational agent system for educational purposes in online courses , 2017, 2017 10th International Conference on Human System Interactions (HSI).

[44]  P. Spooren,et al.  On the Validity of Student Evaluation of Teaching , 2013 .

[45]  Melanie C. Green,et al.  Telephone versus Face-to-Face Interviewing of National Probability Samples with Long Questionnaires: Comparisons of Respondent Satisficing and Social Desirability Response Bias , 2003 .

[46]  Youngme Moon.  Intimate Exchanges: Using Computers to Elicit Self-Disclosure from Consumers , 2000 .

[47]  Clifford Nass,et al.  Computers are social actors , 1994, CHI '94.

[48]  Clint Davies,et al.  Eliciting student feedback for course development: the application of a qualitative course evaluation tool among business research students , 2019 .

[49]  Alexander Maedche,et al.  The Impact of Anthropomorphic and Functional Chatbot Design Features in Enterprise Collaboration Systems on User Acceptance , 2019, Wirtschaftsinformatik.

[50]  J. Krosnick,et al.  Survey research. , 1999, Annual review of psychology.

[51]  Joseph B. Walther,et al.  Selective self-presentation in computer-mediated communication: Hyperpersonal dimensions of technology, language, and cognition , 2007, Comput. Hum. Behav..

[52]  Nicole C. Krämer,et al.  "It doesn't matter what you are!" Explaining social effects of agents and avatars , 2010, Comput. Hum. Behav..

[53]  Matthias Söllner,et al.  Unleashing the Potential of Chatbots in Education: A State-Of-The-Art Analysis , 2018, Academy of Management Proceedings.

[54]  Keri K. Stephens,et al.  R U Able to Meat Me: The Impact of Students’ Overly Casual Email Messages to Instructors , 2009 .

[55]  Joonhwan Lee,et al.  Comparing Data from Chatbot and Web Surveys: Effects of Platform and Conversational Style on Survey Response Quality , 2019, CHI.

[56]  Juan E. Gilbert,et al.  Should AI-Based, conversational digital assistants employ social- or task-oriented interaction style? A task-competency and reciprocity perspective for older adults , 2019, Comput. Hum. Behav..

[57]  Thiemo Wambsganss,et al.  A Conversational Agent to Improve Response Quality in Course Evaluations , 2020, CHI Extended Abstracts.

[58]  Frederick G. Conrad,et al.  Why do survey respondents disclose more when computers ask the questions , 2013 .

[59]  Navneet Kaur,et al.  Opinion mining and sentiment analysis , 2016, 2016 3rd International Conference on Computing for Sustainable Global Development (INDIACom).

[60]  Victoria L. Rubin,et al.  Artificially intelligent conversational agents in libraries , 2010, Libr. Hi Tech.

[61]  Alexander Maedche,et al.  Faster is Not Always Better: Understanding the Effect of Dynamic Response Delays in Human-Chatbot Interaction , 2018, ECIS.

[62]  Susan Bull,et al.  Bringing Chatbots into education: Towards Natural Language Negotiation of Open Learner Models , 2006, SGAI Conf..

[63]  Fang-Wu Tung,et al.  Designing social presence in e-learning environments: Testing the effect of interactivity on children , 2006, Interact. Learn. Environ..

[64]  Detmar W. Straub,et al.  Gender Differences in the Perception and Use of E-Mail: An Extension to the Technology Acceptance Model , 1997, MIS Q..

[65]  Charles Fadel,et al.  Four-Dimensional Education for Sustainable Societies , 2018, Sustainability, Human Well-Being, and the Future of Education.

[66]  David Cameron,et al.  Towards a chatbot for digital counselling , 2017, BCS HCI.

[67]  C. O. Houle.  : Marks of Readable Style: A Study in Adult Education , 1945 .

[68]  Thiemo Wambsganss,et al.  The New Window to Athletes' Soul - What Social Media Tells Us About Athletes' Performances , 2020, HICSS.

[69]  Pushpak Bhattacharyya,et al.  Measuring Sentiment Annotation Complexity of Text , 2014, ACL.

[70]  Thiemo Wambsganss,et al.  Mining User-Generated Repair Instructions from Automotive Web Communities , 2019, HICSS.