Automated, adaptive guidance for K-12 education

Abstract This paper distinguishes features of automated adaptive guidance used in K-12 instructional settings and recommends directions for design. We use meta-analysis to synthesize 24 independent comparisons between automated adaptive guidance and guidance provided during typical teacher-led instruction, and 29 comparisons that isolate the effects of specific adaptive guidance design features in computer-based instruction. We find automated adaptive guidance to be significantly more effective than guidance provided in typical instruction, particularly for students with low prior knowledge. Automated adaptive guidance is most effective when students are generating and integrating ideas (e.g. writing essays, making concept diagrams) as opposed to selecting from the given options. Guidance that promoted self-monitoring was more likely to improve learning outcomes than guidance that addressed only content knowledge. Our findings have implications for researchers who investigate K-12 teaching and learning, designers who create and refine instructional materials using automated guidance, and practitioners who deliver or customize instruction featuring automated guidance.

[1]  Mark Wilson,et al.  Measuring Progressions: Assessment Structures Underlying a Learning Progression , 2009 .

[2]  Beverly Park Woolf,et al.  Evaluation of AnimalWatch: An Intelligent Tutoring System for Arithmetic and Fractions , 2010 .

[3]  K. Koedinger,et al.  Exploring the Assistance Dilemma in Experiments with Cognitive Tutors , 2007 .

[4]  Gwo-Jen Hwang,et al.  Seamless Connection between Learning and Assessment- Applying Progressive Learning Tasks in Mobile Ecology Inquiry , 2013, J. Educ. Technol. Soc..

[5]  K. VanLehn The Relative Effectiveness of Human Tutoring, Intelligent Tutoring Systems, and Other Tutoring Systems , 2011 .

[6]  Wim J. Nijhof,et al.  Effects of complex feedback on computer-assisted modular instruction , 2002, Comput. Educ..

[7]  Gautam Biswas,et al.  Designing Learning by Teaching Agents: The Betty's Brain System , 2008, Int. J. Artif. Intell. Educ..

[8]  Neil T. Heffernan,et al.  A Comparison of Traditional Homework to Computer-Supported Homework , 2009 .

[9]  R. Azevedo Computer Environments as Metacognitive Tools for Enhancing Learning , 2005 .

[10]  P. Black,et al.  Inside the Black Box: Raising Standards through Classroom Assessment , 2010 .

[11]  Susanne Narciss,et al.  Fostering achievement and motivation with bug-related tutoring feedback in a computer-based training for written subtraction. , 2006 .

[12]  S. Vaughn,et al.  A Synthesis of Reading Interventions and Effects on Reading Comprehension Outcomes for Older Struggling Readers , 2009, Review of educational research.

[13]  Kenneth R. Koedinger,et al.  Using Automated Dialog Analysis to Assess Peer Tutoring and Trigger Effective Support , 2011, AIED.

[14]  Daniel C. Moos,et al.  Measuring Cognitive and Metacognitive Regulatory Processes During Hypermedia Learning: Issues and Challenges , 2010 .

[15]  L. Hedges,et al.  Introduction to Meta‐Analysis , 2009, International Coaching Psychology Review.

[16]  Derek C. Briggs,et al.  Experimental and Quasi-Experimental Studies of Inquiry-Based Science Teaching , 2012 .

[17]  Albert T. Corbett,et al.  Why Students Engage in “Gaming the System” Behavior in Interactive Learning Environments , 2008 .

[18]  Andrea A. diSessa,et al.  The Cambridge Handbook of the Learning Sciences: A History of Conceptual Change Research , 2005 .

[19]  John E. Hunter,et al.  Methods of Meta-Analysis: Correcting Error and Bias in Research Findings , 1991 .

[20]  R. Moreno,et al.  Cognitive load and learning effects of having students organize pictures and words in multimedia environments: The role of student interactivity and feedback , 2005 .

[21]  A. Kluger,et al.  The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. , 1996 .

[22]  Kenneth R. Koedinger,et al.  CTRL: A research framework for providing adaptive collaborative learning support , 2009, User Modeling and User-Adapted Interaction.

[23]  Bonnie J. F. Meyer,et al.  Web-Based Tutoring of the Structure Strategy With or Without Elaborated Feedback or Choice for Fifth- and Seventh-Grade Readers , 2010 .

[24]  Neil T. Heffernan,et al.  To Tutor or Not to Tutor: That is the Question , 2009, AIED.

[25]  Yuan-Chen Liu,et al.  Construction of an online learning system for decimal numbers through the use of cognitive conflict strategy , 2008, Comput. Educ..

[26]  Wolfgang Schoppek,et al.  Enhancing Arithmetic and Word-Problem Solving Skills Efficiently by Individualized Computer-Assisted Practice , 2010 .

[27]  Slava Kalyuga,et al.  Rapid dynamic assessment of expertise to improve the efficiency of adaptive e-learning , 2005 .

[28]  David Allbritton,et al.  Tutoring Bilingual Students with an Automated Reading Tutor that Listens , 2007 .

[29]  Ryan Shaun Joazeiro de Baker,et al.  Leveraging machine-learned detectors of systematic inquiry behavior to estimate and predict transfer of inquiry skill , 2011, User Modeling and User-Adapted Interaction.

[30]  R. Bjork Memory and metamemory considerations in the training of human beings. , 1994 .

[31]  Ann L. Brown,et al.  How people learn: Brain, mind, experience, and school. , 1999 .

[32]  Eileen Kintsch,et al.  Summary Street: Interactive Computer Support for Writing , 2004 .

[33]  Beverly Park Woolf,et al.  On-line Tutoring for Math Achievement Testing: A Controlled Evaluation , 2007 .

[34]  Rosemary Luckin,et al.  Designing Adaptive Motivational Scaffolding for a Tutoring System , 2011 .

[35]  L. Hedges,et al.  Statistical Methods for Meta-Analysis , 1987 .

[36]  Jim Minstrell,et al.  Using Questioning to Guide Student Thinking , 1997 .

[37]  Abigail Jurist Levy,et al.  Inquiry-based science instruction—what is it and does it matter? Results from a research synthesis years 1984 to 2002 , 2010 .

[38]  Thomas Bay,et al.  Gaming the system , 2011 .

[39]  Erin Marie Furtak,et al.  Exploring teachers' informal formative assessment practices and students' understanding in the context of scientific inquiry , 2007 .

[40]  Richard E. Mayer,et al.  Polite web-based intelligent tutors: Can they improve learning in classrooms? , 2011, Comput. Educ..

[41]  John R. Anderson,et al.  Cognitive Tutors: Lessons Learned , 1995 .

[42]  Neil T. Heffernan,et al.  A Quasi-Experimental Evaluation of An On-Line Formative Assessment and Tutoring System , 2010 .

[43]  M. Lipsey,et al.  The Way in Which Intervention Studies Have “Personality” and why it is Important to Meta-Analysis , 2001, Evaluation & the health professions.

[44]  Inge Molenaar,et al.  Dynamic scaffolding of socially regulated learning in a computer-based learning environment , 2012, Comput. Educ..

[45]  D. Leutner,et al.  Instructional animation versus static pictures: A meta-analysis , 2007 .

[46]  Leslie R. Herrenkohl,et al.  Pedagogical Practices to Support Classroom Cultures of Scientific Inquiry , 2011 .

[47]  Markus Dresel,et al.  A Computer-Based Approach to Fostering Motivation and Self-Regulated Learning , 2008 .

[48]  Kenneth R. Koedinger,et al.  Designing automated adaptive support to improve student helping behaviors in a peer tutoring activity , 2011, Int. J. Comput. Support. Collab. Learn..

[49]  S G Smith,et al.  Educational uses of the PLATO computer system. , 1976, Science.

[50]  K. Koedinger,et al.  Fostering the Intelligent Novice: Learning From Errors With Metacognitive Tutoring , 2005 .

[51]  Jeffrey D. Karpicke,et al.  Test-Enhanced Learning , 2006, Psychological science.

[52]  Michael D. Steele,et al.  Seeing the Complexity of Standing to the Side: Instructional Dialogues , 2005 .

[53]  Daniel L. Schwartz,et al.  Preparing students for future learning with Teachable Agents , 2010 .

[54]  Marcia C. Linn,et al.  The Science of Learning and the Learning of Science Introducing Desirable Difficulties , 2006 .

[55]  Changhua Sun Rich The Impact of Online Automated Writing Evaluation: A Case Study from Dalian , 2012 .

[56]  Michele W. Spitulnik,et al.  Teacher Use of Evidence to Customize Inquiry Science Instruction. , 2010 .

[57]  Wang Yihong,et al.  Technology-enhanced English Language Writing Assessment in the Classroom* , 2012 .

[58]  Youngkyun Baek,et al.  Improving Recall and Transfer Skills Through Vocabulary Building in Web-Based Second Language Learning: An Examination by Item and Feedback Type , 2008, J. Educ. Technol. Soc..

[59]  Neil T. Heffernan,et al.  Scaffolding vs. Hints in the Assistment System , 2006, Intelligent Tutoring Systems.

[60]  Gautam Biswas,et al.  Modeling and Measuring Self-Regulated Learning in Teachable Agent Environments , 2010, AAAI Fall Symposium: Cognitive and Metacognitive Educational Systems.

[61]  K. VanLehn,et al.  Scaffolding Deep Comprehension Strategies Through Point&Query, AutoTutor, and iSTART , 2005 .

[62]  Anna N. Rafferty,et al.  Promoting Student Learning through Automated Formative Guidance on Chemistry Drawings , 2014, ICLS.

[63]  Paul Ginns Integrating information: A meta-analysis of the spatial contiguity and temporal contiguity effects , 2006 .

[64]  V. Shute Focus on Formative Feedback , 2008 .

[65]  Marcia C. Linn,et al.  An Investigation of Explanation Multiple-Choice Items in Science Assessment , 2011 .

[66]  Russell G. Almond,et al.  You Can't Fatten A Hog by Weighing It - Or Can You? Evaluating an Assessment for Learning System Called ACED , 2008, Int. J. Artif. Intell. Educ..

[67]  Jacob Cohen Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[68]  Peter M. Hastings,et al.  Research Methods Tutor: Evaluation of a dialogue-based tutoring system in the classroom , 2008, Behavior research methods.

[69]  J. Hattie,et al.  The Power of Feedback , 2007 .

[70]  Michael J. Timms Using Item Response Theory (IRT) to select hints in an ITS , 2007, AIED.

[71]  Scott Dooley,et al.  Summary Street®: Computer Support for Comprehension and Writing , 2005 .

[72]  Vincent Aleven,et al.  An effective metacognitive strategy: learning by doing and explaining with a computer-based Cognitive Tutor , 2002, Cogn. Sci..

[73]  Neil T. Heffernan,et al.  Expanding the Model-Tracing Architecture: A 3rd Generation Intelligent Tutor for Algebra Symbolization , 2008, Int. J. Artif. Intell. Educ..

[74]  Gautam Biswas,et al.  The effect of contextualized conversational feedback in a complex open-ended learning environment , 2013 .

[75]  Bonnie J. F. Meyer,et al.  Individualizing a web-based structure strategy intervention for fifth graders' comprehension of nonfiction. , 2011 .

[76]  Lorrie A. Shepard,et al.  Commentary on the National Mathematics Advisory Panel Recommendations on Assessment , 2008 .