A review of automated feedback systems for learners: Classification framework, challenges and opportunities

Abstract Teacher feedback provided to learners in real-time is a crucial factor for their knowledge and skills acquisition. However, providing real-time feedback at an individual level is often infeasible, considering limited teaching resources. Fortunately, recent technological advancements have allowed for developing of various computer tutoring systems, which can support learners at any place and time by generating personalized feedback automatically. Such systems have emerged in various domains, tackle different educational tasks, and often are designed in very distinctive ways. Consequently, the knowledge in the field of automated feedback systems is rather scattered across different domains and applications, as we illustrate in this study. This paper aims to outline the state-of-the-art of recently developed systems for delivering automated feedback, and thus serves as a source of systematic information for educators, researchers in the educational domain and system developers. More specifically, the contribution of this study is twofold. Firstly, we offer an extensive literature review of the field. As a result of a rigorous selection process, consisting of 4 phases, a total of 109 automated feedback systems is selected for a detailed review and thoroughly classified against numerous dimensions. Secondly, based on a detailed analysis of the recent literature sources and following the design science research approach, a classification framework for automated feedback systems is developed. This framework is used to classify the selected systems, and thus give a detailed overview of the predominantly available educational technologies, the educational settings in which they are applied, the properties of automated feedback they deliver, and the approaches for their design and evaluation. Based on this analysis, several important observations and recommendations are put forward as an outcome of the study. In particular, our study outlines the current fragmentation of the field, discusses a need for a common reference framework, and calls for more personalized, data-driven and student-centered solutions to exploit a larger set of opportunities offered in this age of data.

[1]  Kurt VanLehn,et al.  Learning How to Construct Models of Dynamic Systems: An Initial Evaluation of the Dragoon Intelligent Tutoring System , 2017, IEEE Transactions on Learning Technologies.

[2]  Jesualdo Tomás Fernández-Breis,et al.  An extension of the OeLE platform for generating semantic feedback for students and teachers , 2010 .

[3]  Claus Zinn.  Algorithmic Debugging and Literate Programming to Generate Feedback in Intelligent Tutoring Systems , 2014, KI.

[4]  K. Koedinger,et al.  Improving students’ help-seeking skills using metacognitive feedback in an intelligent tutoring system , 2011, Learning and Instruction.

[5]  V. Shute.  Focus on Formative Feedback , 2008 .

[6]  Sahana Murthy,et al.  Personalized Affective Feedback to Address Students’ Frustration in ITS , 2019, IEEE Transactions on Learning Technologies.

[7]  Glenn D. Blank,et al.  Individualizing Tutoring with Learning Style Based Feedback , 2008, Intelligent Tutoring Systems.

[8]  Rafael A. Calvo,et al.  Improving Medical Students’ Awareness of Their Non-Verbal Communication through Automated Non-Verbal Behavior Feedback , 2016, Front. ICT.

[9]  Gimeno Sanz Ana,et al.  Designing feedback to support language acquisition using the ingenio authoring tool , 2009 .

[10]  D. Cook,et al.  Computerized Virtual Patients in Health Professions Education: A Systematic Review and Meta-Analysis , 2010, Academic medicine : journal of the Association of American Medical Colleges.

[11]  Anita R. Bowles,et al.  Technologies for foreign language learning: a review of technology types and their effectiveness , 2014 .

[12]  Olga C. Santos,et al.  Gui-driven intelligent tutoring system with affective support to help learning the algebraic method , 2017, 2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC).

[13]  Taro Fujita,et al.  Learners’ use of domain-specific computer-based feedback to overcome logical circularity in deductive proving in geometry , 2018 .

[14]  Evandro Costa,et al.  An Adaptive Approach to Provide Feedback for Students in Programming Problem Solving , 2019, ITS.

[15]  Claes Wohlin,et al.  Guidelines for snowballing in systematic literature studies and a replication in software engineering , 2014, EASE '14.

[16]  Kaoru Sumi,et al.  Adaptive Feedback Based on Student Emotion in a System for Programming Practice , 2018, ITS.

[17]  G.W.M. Rauterberg,et al.  Online feedback system for public speakers , 2012, 2012 IEEE Symposium on E-Learning, E-Management and E-Services.

[18]  John Atkinson,et al.  Adaptive feedback selection for intelligent tutoring systems , 2011, Expert Syst. Appl..

[19]  Monique Snoeck,et al.  Conceptual framework for feedback automation in SLEs , 2016 .

[20]  James D. Klein,et al.  Control of feedback in computer-assisted instruction , 1991 .

[21]  Sumit Gulwani,et al.  Automated feedback generation for introductory programming assignments , 2012, ACM-SIGPLAN Symposium on Programming Language Design and Implementation.

[22]  David Bañeres,et al.  Intelligent Tutoring System for Learning Digital Systems on MOOC Environments , 2016, 2016 10th International Conference on Complex, Intelligent, and Software Intensive Systems (CISIS).

[23]  Vimala Balakrishnan,et al.  Identification of personal traits in adaptive learning environment: Systematic literature review , 2019, Comput. Educ..

[24]  Vahid R. Mirzaeian,et al.  Learning Persian grammar with the aid of an intelligent feedback generator , 2016, Eng. Appl. Artif. Intell..

[25]  Christof Lutteroth,et al.  glGetFeedback — Towards automatic feedback and assessment for OpenGL 3D modelling assignments , 2016, 2016 International Conference on Image and Vision Computing New Zealand (IVCNZ).

[26]  Eric Ras,et al.  Designing Formative and Adaptive Feedback Using Incremental User Models , 2016, ICWL.

[27]  Davide Fossati,et al.  Simple but effective feedback generation to tutor abstract problem solving , 2008, INLG.

[28]  Tracy Anne Hammond,et al.  Maestoso: An Intelligent Educational Sketching Tool for Learning Music Theory , 2015, AAAI.

[29]  Sandra Katz,et al.  Eliciting student explanations during tutorial dialogue for the purpose of providing formative feedback , 2013, AIED Workshops.

[30]  Dror Ben-Naim,et al.  Instructional Support for Teachers and Guided Feedback for Students in an Adaptive eLearning Environment , 2011, 2011 Eighth International Conference on Information Technology: New Generations.

[31]  Charles E. Hughes,et al.  Providing Real-time Feedback for Student Teachers in a Virtual Rehearsal Environment , 2015, ICMI.

[32]  Jody Oomen-Early,et al.  Personalized Versus Collective Instructor Feedback in the Online Courseroom: Does Type of Feedback , 2008 .

[33]  Indira Thouvenin,et al.  Adaptive Training Environment without Prior Knowledge: Modeling Feedback Selection as a Multi-armed Bandit Problem , 2016, UMAP.

[34]  Stephen B. Gilbert,et al.  AC 2008-1746: INTEGRATION OF AN INTELLIGENT TUTORING SYSTEM WITH A WEB-BASED AUTHORING SYSTEM TO DEVELOP ONLINE HOMEWORK ASSIGNMENTS WITH FORMATIVE FEEDBACK , 2008 .

[35]  Zachary T. Chung,et al.  An Intelligent Tutoring System for Japanese Language Particles with User Assessment and Feedback , 2013, AIED Workshops.

[36]  Ke Wang,et al.  Data-Driven Feedback Generator for Online Programing Courses , 2017, L@S.

[37]  Erwin Fielt,et al.  European Conference on Information Systems ( ECIS ) Summer 10-6-2011 A SYSTEMATIC , TOOL-SUPPORTED METHOD FOR CONDUCTING LITERATURE REVIEWS IN INFORMATION SYSTEMS , 2017 .

[38]  Mengxiao Zhu,et al.  Investigating the impact of automated feedback on students’ scientific argumentation , 2017 .

[39]  Jan Elen,et al.  Conceptualizing the Domain of Automated Feedback for Learners , 2019, CIbSE.

[40]  Manolis Mavrikis,et al.  Affective learning: improving engagement and enhancing learning with affect-aware feedback , 2017, User Modeling and User-Adapted Interaction.

[41]  Ling Guan,et al.  An Approach to Ballet Dance Training through MS Kinect and Visualization in a CAVE Virtual Reality Environment , 2015, ACM Trans. Intell. Syst. Technol..

[42]  Peter van Rosmalen,et al.  Can You Help Me with My Pitch? Studying a Tool for Real-Time Automated Feedback , 2016, IEEE Transactions on Learning Technologies.

[43]  Christopher Cheong,et al.  Improving Quality of Feedback Using a Technology-Supported Learning System , 2015, PACIS.

[44]  Mohammed E. Hoque,et al.  ROC speak: semi-automated personalized feedback on nonverbal behavior from recorded videos , 2015, UbiComp.

[45]  Norisma Idris,et al.  Adaptive feedback in computer-based learning environments: a review , 2017, Adapt. Behav..

[46]  Rynson W. H. Lau,et al.  Learning Programming Languages through Corrective Feedback and Concept Visualisation , 2011, ICWL.

[47]  Ioannis Hatzilygeroudis,et al.  Assistance and Feedback Mechanism in an Intelligent Tutoring System for Teaching Conversion of Natural Language into Logic , 2017, International Journal of Artificial Intelligence in Education.

[48]  Khaled Shaalan,et al.  Analysis and feedback of erroneous Arabic verbs , 2013, Natural Language Engineering.

[49]  Johan Jeuring,et al.  A Systematic Literature Review of Automated Feedback Generation for Programming Exercises , 2018, ACM Trans. Comput. Educ..

[50]  Kia Ng.  Interactive Multimedia for Technology-Enhanced Learning with Multimodal Feedback , 2011 .

[51]  Trevor Barker.  An Automated Feedback System Based on Adaptive Testing: Extending the Model , 2010, iJET.

[52]  Susanne P. Lajoie,et al.  Motivation and emotion predict medical students’ attention to computer-based feedback , 2017, Advances in Health Sciences Education.

[53]  Carl D. Westine,et al.  Systematic review of adaptive learning research designs, context, strategies, and technologies from 2009 to 2018 , 2020 .

[54]  Rafael A. Calvo,et al.  Students' Conceptions of Tutor and Automated Feedback in Professional Writing , 2010 .

[55]  Thomas W. Price,et al.  Generating Data-driven Hints for Open-ended Programming , 2016, EDM.

[56]  Margus Niitsoo,et al.  MatchMySound: Introducing Feedback to Online Music Education , 2014, ICWL Workshops.

[57]  B. Nitish,et al.  Enhancing JFLAP with automata construction problems and automated feedback , 2014, 2014 Seventh International Conference on Contemporary Computing (IC3).

[58]  Luciana Benotti,et al.  The Effect of a Web-based Coding Tool with Automatic Feedback on Students' Performance and Perceptions , 2018, SIGCSE.

[59]  Anastasios A. Economides,et al.  Personalized Feedback in CAT , 2005 .

[60]  Danielle S. McNamara,et al.  Feedback and Revising in an Intelligent Tutoring System for Writing Strategies , 2013, AIED.

[61]  D. Baker,et al.  Early science learning with a virtual tutor through multimedia explanations and feedback on spoken questions , 2018 .

[62]  Johanna D. Moore,et al.  BEETLE II: Deep Natural Language Understanding and Automatic Feedback Generation for Intelligent Tutoring in Basic Electricity and Electronics , 2014, International Journal of Artificial Intelligence in Education.

[63]  Nadia Mana,et al.  Interactive stories and exercises with dynamic feedback for improving reading comprehension skills in deaf children , 2013, Comput. Educ..

[64]  Davide Fossati,et al.  Data D riven A utomatic F eedback G eneration in the iList I ntelligent T utoring S ystem , 2014 .

[65]  Lilia Cheniti Belcadhi.  Personalized feedback for self assessment in lifelong learning environments based on semantic web , 2016 .

[66]  Benedict du Boulay,et al.  Motivational and metacognitive feedback in SQL-Tutor* , 2015, Comput. Sci. Educ..

[67]  Siriwan Suebnukarn,et al.  Intelligent dental training simulator with objective skill assessment and feedback , 2011, Artif. Intell. Medicine.

[68]  James R. Curran,et al.  Data Mining to Generate Individualised Feedback , 2010, Intelligent Tutoring Systems.

[69]  Julio C. Caiza,et al.  PROGRAMMING ASSIGNMENTS AUTOMATIC GRADING: REVIEW OF TOOLS AND IMPLEMENTATIONS , 2013 .

[70]  Jonathan Gratch,et al.  Autonomous Agent that Provides Automated Feedback Improves Negotiation Skills , 2018, AIED.

[71]  Gregor Kennedy,et al.  Mining interactions in immersive learning environments for real-time student feedback , 2013 .

[72]  Alan R. Hevner,et al.  Design Science in Information Systems Research , 2004, MIS Q..

[73]  Siriwan Suebnukarn,et al.  Employing UMLS for generating hints in a tutoring system for medical problem-based learning , 2012, J. Biomed. Informatics.

[74]  Mika Luimula,et al.  Higher Education Learner Experience with Fuzzy Feedback in a Digital Learning Environment , 2018, 2018 9th IEEE International Conference on Cognitive Infocommunications (CogInfoCom).

[75]  David Arnau,et al.  Domain-specific knowledge representation and inference engine for an intelligent tutoring system , 2013, Knowl. Based Syst..

[76]  Elaine Toms,et al.  What is user engagement? A conceptual framework for defining user engagement with technology , 2008, J. Assoc. Inf. Sci. Technol..

[77]  D. Boud,et al.  Rethinking models of feedback for learning: the challenge of design , 2013 .

[78]  Jesualdo Tomás Fernández-Breis,et al.  Semantic Web technologies for generating feedback in online assessment environments , 2012, Knowl. Based Syst..

[79]  Sheng-Jen Hsieh,et al.  An intelligent tutoring system for computer numerical control programming , 2019 .

[80]  Helen J. Parkin,et al.  Using technology to encourage student engagement with feedback: a literature review , 2011 .

[81]  Kalina Yacef,et al.  Forró Trainer: Automated Feedback for Partner Dance Learning , 2017, UMAP.

[82]  Paulien C. Meijer,et al.  Types and frequencies of feedback interventions in classroom interaction in secondary education , 2012 .

[83]  Stefan Kopp,et al.  Classification of motor errors to provide real-time feedback for sports coaching in virtual reality - A case study in squats and Tai Chi pushes , 2018, Comput. Graph..

[84]  Cumhur Erkut,et al.  Auditory feedback in an interactive rhythmic tutoring system , 2011, AM '11.

[85]  Davide Fossati,et al.  Be Brief, And They Shall Learn: Generating Concise Language Feedback for a Computer Tutor , 2008, Int. J. Artif. Intell. Educ..

[86]  Zacharias C. Zacharia,et al.  Identifying potential types of guidance for supporting student inquiry when using virtual and remote labs in science: a literature review , 2015, Educational Technology Research and Development.

[87]  R. Monty,et al.  The importance of perceived control: fact or fantasy? , 1977, American scientist.

[88]  Paul Salvador Inventado,et al.  Predicting Student's Appraisal of Feedback in an ITS Using Previous Affective States and Continuous Affect Labels from EEG Data , 2010 .

[89]  Daniel L. Schwartz,et al.  Assessing Whether Students Seek Constructive Criticism: The Design of an Automated Feedback System for a Graphic Design Task , 2017, International Journal of Artificial Intelligence in Education.

[90]  S. Narciss.  Feedback Strategies for Interactive Learning Tasks , 2007 .

[91]  Scotty D. Craig,et al.  Exploring the effectiveness of a novel feedback mechanism within an intelligent tutoring system , 2015, Int. J. Learn. Technol..

[92]  Daniel Kelly,et al.  A system for teaching sign language using live gesture feedback , 2008, 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition.

[93]  Antonija Mitrovic,et al.  The effect of positive feedback in a constraint-based intelligent tutoring system , 2013, Comput. Educ..

[94]  Amy Pallant,et al.  Automated text scoring and real‐time adjustable feedback: Supporting revision of scientific arguments involving uncertainty , 2019, Science Education.

[95]  Gwo-Jen Hwang,et al.  Definition, framework and research issues of smart learning environments - a context-aware ubiquitous learning perspective , 2014, Smart Learning Environments.

[96]  Helmer Strik,et al.  Spoken grammar practice and feedback in an ASR-based CALL system , 2015 .

[97]  Xavier Ochoa,et al.  The RAP system: automatic feedback of oral presentation skills using multimodal analysis and low-cost sensors , 2018, LAK.

[98]  Wayne H. Ward,et al.  My Science Tutor: A Conversational Multimedia Virtual Tutor. , 2013 .

[99]  Korinn S. Ostrow,et al.  Counterintuitive effects of online feedback in middle school math: results from a randomized controlled trial in ASSISTments , 2017 .

[100]  Gary M Velan,et al.  Knowledge maps: a tool for online assessment with automated feedback , 2018, Medical education online.

[101]  James Daniel Lehman,et al.  Using Targeted Feedback to Address Common Student Misconceptions in Introductory Programming: A Data-Driven Approach , 2019, SAGE Open.

[102]  K. VanLehn.  The Relative Effectiveness of Human Tutoring, Intelligent Tutoring Systems, and Other Tutoring Systems , 2011 .

[103]  Lindsay Alexander Shaw,et al.  CodeRunnerGL - An Interactive Web-Based Tool for Computer Graphics Teaching and Assessment , 2019, 2019 International Conference on Electronics, Information, and Communication (ICEIC).

[104]  Monique Snoeck,et al.  Assessing the influence of feedback-inclusive rapid prototyping on understanding the semantics of parallel UML statecharts by novice modellers , 2017, Inf. Softw. Technol..

[105]  Mohamed Medhat Gaber,et al.  OntoPeFeGe: Ontology-Based Personalized Feedback Generator , 2018, IEEE Access.

[106]  Andrew Stranieri,et al.  Enhancing learning outcomes with an interactive knowledge-based learning environment providing narrative feedback , 2008, Interact. Learn. Environ..

[107]  Manolis Mavrikis,et al.  Feedback Authoring for Exploratory Activities: The Case of a Logo-Based 3D Microworld , 2016, CSEDU.

[108]  Thapanee Seechaliao.  Instructional Strategies to Support Creativity and Innovation in Education. , 2017 .

[109]  Dragan Gasevic,et al.  Using learning analytics to scale the provision of personalised feedback , 2019, Br. J. Educ. Technol..

[110]  Siriwan Suebnukarn,et al.  Leveraging a Domain Ontology to Increase the Quality of Feedback in an Intelligent Tutoring System , 2010, Intelligent Tutoring Systems.

[111]  S. Chatterjee,et al.  Design Science Research in Information Systems , 2010 .

[112]  Niels Pinkwart,et al.  A Review of AI-Supported Tutoring Approaches for Learning Programming , 2013, Advanced Computational Methods for Knowledge Engineering.

[113]  Froduald Kabanza,et al.  Delivering Tutoring Feedback Using Persuasive Dialogues , 2010, Intelligent Tutoring Systems.

[114]  Taku Komura,et al.  A Virtual Reality Dance Training System Using Motion Capture Technology , 2011, IEEE Transactions on Learning Technologies.

[115]  Ming Liu,et al.  Automated Essay Feedback Generation and Its Impact on Revision , 2017, IEEE Transactions on Learning Technologies.

[116]  B. Bloom.  The 2 Sigma Problem: The Search for Methods of Group Instruction as Effective as One-to-One Tutoring , 1984 .

[117]  Hejie Chen,et al.  A Framework of Deriving Adaptive Feedback from Educational Ontologies , 2008, 2008 The 9th International Conference for Young Computer Scientists.

[118]  Inn-Chull Choi,et al.  Efficacy of an ICALL tutoring system and process-oriented corrective feedback , 2016 .

[119]  Shuo Zhou,et al.  Providing Effective Real-Time Feedback in Simulation-Based Surgical Training , 2017, MICCAI.

[120]  Reyes Juárez-Ramírez,et al.  A feedback system to provide affective support to students , 2018, Comput. Appl. Eng. Educ..

[121]  Hendrik Drachsler,et al.  Dancing Salsa with Machines—Filling the Gap of Dancing Learning Solutions , 2019, Sensors.

[122]  Peter Ferguson,et al.  Student perceptions of quality feedback in teacher education , 2011 .

[123]  Engelbert Mephu Nguifo,et al.  Multi-paradigm Generation of Tutoring Feedback in Robotic Arm Manipulation Training , 2012, ITS.

[124]  Haiyang Ai.  Providing graduated corrective feedback in an intelligent computer-assisted language learning environment , 2017, ReCALL.

[125]  Petri Ihantola,et al.  A mobile learning application for parsons problems with automatic feedback , 2012, Koli Calling.

[126]  Joshua Wilson,et al.  Automated essay evaluation software in English Language Arts classrooms: Effects on teacher feedback, student motivation, and writing quality , 2016, Comput. Educ..

[127]  J. Sun,et al.  Effects of intelligent feedback on online learners’ engagement and cognitive load: the case of research ethics education , 2018, Educational Psychology.

[128]  Johan Jeuring,et al.  Ask-Elle: an Adaptable Programming Tutor for Haskell Giving Automated Feedback , 2017, International Journal of Artificial Intelligence in Education.

[129]  Diane D. Chapman,et al.  Automating Individualized Formative Feedback in Large Classes Based on a Directed Concept Graph , 2017, Front. Psychol..

[130]  J. Hattie,et al.  The Power of Feedback , 2007 .

[131]  Eric Andres,et al.  Exploring feedback and student characteristics relevant for personalizing feedback strategies , 2014, Comput. Educ..

[132]  Kenneth R. Koedinger,et al.  Automatic Generation of Programming Feedback; A Data-Driven Approach , 2013, AIED Workshops.

[133]  Gautam Biswas,et al.  Learner modeling for adaptive scaffolding in a Computational Thinking-based science learning environment , 2017, User Modeling and User-Adapted Interaction.

[134]  Gary Cheng,et al.  The impact of online automated feedback on students' reflective journal writing in an EFL course , 2017, Internet High. Educ..

[135]  Yao-Ting Sung,et al.  The effect of online summary assessment and feedback system on the summary writing on 6th graders: The LSA-based technique , 2016, Comput. Educ..

[136]  Burkhard Wünsche,et al.  Technologies and Tools to Support Teaching and Learning Computer Graphics: A Literature Review , 2019, ACE '19.

[137]  Gökhan Akçapinar,et al.  How automated feedback through text mining changes plagiaristic behavior in online assignments , 2015, Comput. Educ..

[138]  Sergio Salmeron-Majadas,et al.  Some insights into the impact of affective information when delivering feedback to students , 2018, Behav. Inf. Technol..