A Framework to support the Experimental Evaluation Process of the Pedagogical Conversational Systems

Conversational systems are applications whose main characteristic is interaction with humans through spoken or written natural language. Initially, they have been established for educational purposes, dedicated to supporting students while learning new subjects, motivating them while carrying out activities, solving their questions, indicating new learning materials, among others. Several studies show the benefits of using conversational systems how a support tool, promoting the application of active learning methodologies. However, recent studies have mentioned that these systems are not being adequately evaluated since it is not possible to observe systematization in the evaluations, the selection of variables is not standardized, and there is no replicated study yet. Experimental Software Engineering presents robust procedures for experimentation. It can indicate better strategies to evaluate this type of system and offer subsidies to improve the visibility of the potential of these systems as a learning support mechanism. In this project, we intend to collaborate whit the evaluation process of the conversational systems, proposing a framework to support the planning, execution, and reporting of the experimental studies. The area lacks adequate methodologies for reliable experimental studies and, therefore, this project can contribute to improving the quality of conversational systems.

[1]  Guilherme Horta Travassos,et al.  Experimentation with dynamic simulation models in software engineering: planning and reporting guidelines , 2015, Empirical Software Engineering.

[2]  Shari Lawrence Pfleeger,et al.  Preliminary Guidelines for Empirical Research in Software Engineering , 2002, IEEE Trans. Software Eng..

[3]  Judithe Sheard,et al.  Analysis of research into the teaching and learning of programming , 2009, ICER '09.

[4]  Guilherme Horta Travassos,et al.  Introdução à Engenharia de Software Experimental , 2002 .

[5]  Matthias Söllner,et al.  Unleashing the Potential of Chatbots in Education: A State-Of-The-Art Analysis , 2018, Academy of Management Proceedings.

[6]  Forrest Shull,et al.  Building Knowledge through Families of Experiments , 1999, IEEE Trans. Software Eng..

[7]  Carlos Delgado Kloos,et al.  Supporting a MOOC through a Conversational Agent. Design of a First Prototype , 2018, 2018 International Symposium on Computers in Education (SIIE).

[8]  H. N. Io,et al.  Chatbots and conversational agents: A bibliometric analysis , 2017, 2017 IEEE International Conference on Industrial Engineering and Engineering Management (IEEM).

[9]  Shwetak N. Patel,et al.  Evaluating and Informing the Design of Chatbots , 2018, Conference on Designing Interactive Systems.

[10]  Sebastian Hobert,et al.  How Are You, Chatbot? Evaluating Chatbots in Educational Settings - Results of a Literature Review , 2019, DeLFI.

[11]  Paul J. Fortier,et al.  Computer Systems Performance Evaluation and Prediction , 2003 .

[12]  Leo Natan Paschoal,et al.  A Chatterbot Sensitive to Student's Context to Help on Software Engineering Education , 2018, 2018 XLIV Latin American Computer Conference (CLEI).

[13]  Cristiano André da Costa,et al.  Survey of conversational agents in health , 2019, Expert Syst. Appl..

[14]  Jessica A. Chen,et al.  Conversational agents in healthcare: a systematic review , 2018, J. Am. Medical Informatics Assoc..

[15]  Anbang Xu,et al.  A New Chatbot for Customer Service on Social Media , 2017, CHI.

[16]  Paul J. Fortier,et al.  11 – System Performance Evaluation Tool Selection and Use , 2003 .

[17]  Jeffrey C. Carver,et al.  A Pragmatic Documents Standard for an Experience Library: Roles,Documen, Contents and Structure , 2001 .

[18]  Aliane Loureiro Krassmann,et al.  Conversational Agents in Distance Education: Comparing Mood States with Students’ Perception , 2018 .

[19]  John Woods,et al.  Survey on Chatbot Design Techniques in Speech Conversation Systems , 2015 .

[20]  Guilherme Horta Travassos,et al.  An Environment to Support Large Scale Experimentation in Software Engineering , 2008, 13th IEEE International Conference on Engineering of Complex Computer Systems (iceccs 2008).

[21]  Jeffrey C. Carver,et al.  An empirical methodology for introducing software processes , 2001, ESEC/FSE-9.

[22]  Claes Wohlin,et al.  Experimentation in Software Engineering , 2000, The Kluwer International Series in Software Engineering.

[23]  Beatriz Marín,et al.  A Methodological Framework for Evaluating Software Testing Techniques and Tools , 2012, 2012 12th International Conference on Quality Software.

[24]  Simone do Rocio Senger de Souza,et al.  Towards a Conversational Agent to Support the Software Testing Education , 2019, SBES.