Combined inner and outer loop feedback in an intelligent tutoring system for statistics in higher education

Peer Review The peer review history for this article is available at https://publons.com/publon/10. 1111/jcal.12491. Abstract Intelligent tutoring systems (ITSs) can provide inner loop feedback about steps within tasks, and outer loop feedback about performance on multiple tasks. While research typically addresses these feedback types separately, many ITSs offer them simultaneously. This study evaluates the effects of providing combined inner and outer loop feedback on social sciences students' learning process and performance in a first-year university statistics course. In a 2 x 2 factorial design (elaborate inner loop vs. minimal inner loop and outer loop vs. no outer loop feedback) with 521 participants, the effects of both feedback types and their combination were assessed through multiple linear regression models. Results showed mixed effects, depending on students' prior knowledge and experience, and no overall effects on course performance. Students tended to use outer loop feedback less when also receiving elaborate inner loop feedback. We therefore recommend introducing feedback types one by one and offering them for substantial periods of time.

[1]  Vladimir Zadorozhny,et al.  The Value of Social: Comparing Open Student Modeling and Open Social Student Modeling , 2015, UMAP.

[2]  K. VanLehn.  The Relative Effectiveness of Human Tutoring, Intelligent Tutoring Systems, and Other Tutoring Systems , 2011 .

[3]  Gillian Lancaster,et al.  Statistical Education in the 21st Century: A Review of Challenges, Teaching Innovations and Strategies for Reform , 2012 .

[4]  Judy Kay,et al.  SMILI☺: a Framework for Interfaces to Learning Data in Open Learner Models, Learning Analytics and Related Fields , 2016, International Journal of Artificial Intelligence in Education.

[5]  J. Hattie,et al.  The Power of Feedback , 2007 .

[6]  Eric Andres,et al.  Exploring feedback and student characteristics relevant for personalizing feedback strategies , 2014, Comput. Educ..

[7]  Carrie Demmans Epp,et al.  Evaluating the Effect of Uncertainty Visualisation in Open Learner Models on Students' Metacognitive Skills , 2017, AIED.

[8]  John R. Anderson,et al.  Cognitive Tutors: Lessons Learned , 1995 .

[9]  R. Falk,et al.  Significance Tests Die Hard , 1995 .

[10]  Caroline F. Timmers,et al.  Motivational beliefs, student effort, and feedback behaviour in computer-based formative assessment , 2013, Comput. Educ..

[11]  Vincent Aleven,et al.  Students' Understanding of Their Student Model , 2011, AIED.

[12]  J. Braak,et al.  The Gap between Educational Research and Practice: Views of Teachers, School Leaders, Intermediaries and Researchers. , 2010 .

[13]  Peter Brusilovsky,et al.  Evaluation of topic-based adaptation and student modeling in QuizGuide , 2015, User Modeling and User-Adapted Interaction.

[14]  Johan Jeuring,et al.  Automated Feedback on the Structure of Hypothesis Tests , 2019, AIED.

[15]  Wim Van Den Noortgate,et al.  Students’ misconceptions of statistical inference: A review of the empirical evidence from research on statistics education , 2007 .

[16]  V. Shute.  Focus on Formative Feedback , 2008 .

[17]  C. Evans.  Making Sense of Assessment Feedback in Higher Education , 2013 .

[18]  Larry Ambrose,et al.  The power of feedback. , 2002, Healthcare executive.

[19]  Antonija Mitrovic,et al.  Intelligent Tutors for All: The Constraint-Based Approach , 2007, IEEE Intelligent Systems.

[20]  Joan Garfield,et al.  The Statistics Teaching Inventory: A Survey on Statistics Teachers' Classroom Practices and Beliefs , 2012 .

[21]  Beverly Park Woolf,et al.  Repairing Disengagement With Non-Invasive Interventions , 2007, AIED.

[22]  Antonija Mitrovic,et al.  Evaluating the Effect of Open Student Models on Self-Assessment , 2007, Int. J. Artif. Intell. Educ..

[23]  Susanne Narciss,et al.  How to design informative tutoring feedback for multi-media learning , 2004 .

[24]  K. Taber.  The Use of Cronbach’s Alpha When Developing and Reporting Research Instruments in Science Education , 2017, Research in Science Education.

[25]  Henry May,et al.  Rethinking Connections Between Research and Practice in Education: A Conceptual Framework , 2018 .

[26]  Abelardo Pardo,et al.  A feedback model for data-rich learning experiences , 2018 .

[27]  J. W. Gikandi,et al.  Online formative assessment in higher education: A review of the literature , 2011, Comput. Educ..

[28]  Paul Drijvers,et al.  The Interplay between Inspectable Student Models and Didactics of Statistics , 2018 .

[29]  Peter Brusilovsky,et al.  User Models for Adaptive Hypermedia and Adaptive Educational Systems , 2007, The Adaptive Web.

[30]  Paul Drijvers,et al.  Author response for "Combined inner and outer loop feedback in an intelligent tutoring system for statistics in higher education" , 2020 .

[31]  Joaquim A. Jorge,et al.  Interoperable Intelligent Tutoring Systems as Open Educational Resources , 2013, IEEE Transactions on Learning Technologies.

[32]  Fabienne M. Van der Kleij,et al.  Effects of Feedback in a Computer-Based Learning Environment on Students’ Learning Outcomes , 2013 .

[33]  Giorgi Goguadze.  ActiveMath - generation and reuse of interactive exercises using domain reasoners and automated tutorial strategies , 2010 .

[34]  Joan Garfield,et al.  Learning to Reason About Statistical Inference , 2008 .

[35]  Kurt VanLehn,et al.  The Behavior of Tutoring Systems , 2006, Int. J. Artif. Intell. Educ..

[36]  David Rosenthal,et al.  An Assessment of Constraint-Based Tutors: A Response to Mitrovic and Ohlsson's Critique of "A Comparison of Model-Tracing and Constraint-Based Intelligent Tutoring Paradigms" , 2006, Int. J. Artif. Intell. Educ..

[37]  A. Kluger,et al.  The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. , 1996 .

[38]  D. Nicol,et al.  Formative assessment and self‐regulated learning: a model and seven principles of good feedback practice , 2006 .

[39]  Paul Drijvers,et al.  Digital design: RME principles for designing online tasks , 2013 .