Guided Retrieval Practice of Educational Materials Using Automated Scoring

Retrieval practice is a powerful way to promote long-term retention and meaningful learning. However, students do not frequently practice retrieval on their own, and when they do, they have difficulty evaluating the correctness of their responses and making effective study choices. To address these problems, we have developed a guided retrieval practice program that uses an automated scoring algorithm, called QuickScore, to evaluate responses during retrieval practice and make study choices based on student performance. In Experiments 1A and 1B, students learned human anatomy materials in either repeated retrieval or repeated study conditions. Repeated retrieval in the computer-based program produced large gains in retention on a delayed test. In Experiment 2, we examined the accuracy of QuickScore’s scoring relative to students’ self-scoring of their own responses. Students exhibited a dramatic bias to give partial or full credit to completely incorrect responses, while QuickScore was far less likely to score incorrect responses as correct. These results support the efficacy of computer guided retrieval practice for promoting long-term learning.

[1]  Jeffrey D. Karpicke,et al.  Metacognitive strategies in student learning: Do students practise retrieval when they study on their own? , 2009, Memory.

[2]  John Dunlosky,et al.  Improving college students’ evaluation of text learning using idea-unit standards , 2011, Quarterly journal of experimental psychology.

[3]  Danielle S. McNamara,et al.  Using LSA in AutoTutor: Learning Through Mixed-Initiative Dialogue in Natural Language , 2007 .

[4]  Lisa K. Son,et al.  Learners’ choices and beliefs about self-testing , 2009, Memory.

[5]  Katherine A. Rawson,et al.  Why Testing Improves Memory: Mediator Effectiveness Hypothesis , 2010, Science.

[6]  J. Fleiss Statistical methods for rates and proportions , 1974 .

[7]  Jeffrey D. Karpicke,et al.  The Critical Importance of Retrieval for Learning , 2008, Science.

[8]  John Dunlosky,et al.  Improving students’ self-evaluation of learning for key concepts in textbook materials , 2007 .

[9]  Jeffrey D. Karpicke,et al.  Retrieval-Based Learning: A Perspective for Enhancing Meaningful Learning , 2012 .

[10]  Arthur C. Graesser,et al.  Using LSA in AutoTutor: Learning through mixed-initiative dialogue in natural language. , 2007 .

[11]  George A. Miller,et al.  WordNet: A Lexical Database for English , 1995, HLT.

[12]  A A Zaia,et al.  A simple method for the analysis of root canal preparation. , 2000, Journal of endodontics.

[13]  Trudy W. Banta,et al.  Promise and perils , 1988 .

[14]  Jeffrey D. Karpicke,et al.  Test-Enhanced Learning , 2006, Psychological science.

[15]  Kevin C. Almeroth,et al.  Clickers in college classrooms: Fostering learning with questioning methods in large lecture classes , 2009 .

[16]  Christiane Fellbaum,et al.  Book Reviews: WordNet: An Electronic Lexical Database , 1999, CL.

[17]  Henry L. Roediger,et al.  Is expanding retrieval a superior method for learning text materials? , 2010, Memory & cognition.

[18]  Danielle S. McNamara,et al.  Evaluating Self-Explanations in iSTART , 2007 .

[19]  Arthur C. Graesser,et al.  Select-a-Kibitzer: A Computer Tool that Gives Meaningful Feedback on Student Compositions , 2000, Interact. Learn. Environ..

[20]  Martin F. Porter,et al.  An algorithm for suffix stripping , 1997, Program.

[21]  Katherine A. Rawson,et al.  Testing the retrieval effort hypothesis: Does greater difficulty correctly recalling information lead to higher levels of memory? , 2009 .

[22]  Jeffrey D. Karpicke,et al.  Spaced retrieval: absolute spacing enhances learning regardless of relative spacing. , 2011, Journal of experimental psychology. Learning, memory, and cognition.

[23]  Henry L. Roediger,et al.  Examining the Testing Effect with Open-and Closed-book Tests , 2022 .

[24]  Jeffrey D. Karpicke,et al.  Retrieval Practice Produces More Learning than Elaborative Studying with Concept Mapping , 2011, Science.

[25]  Elijah Mayfield,et al.  Transforming Biology Assessment with Machine Learning: Automated Scoring of Written Evolutionary Explanations , 2012 .

[26]  Jeffrey D. Karpicke Retrieval-Based Learning , 2012 .

[27]  Cheryl I. Johnson,et al.  A Testing Effect with Multimedia Learning , 2009 .

[28]  John Dunlosky,et al.  Overconfidence produces underachievement: Inaccurate self evaluations undermine students’ learning and retention , 2012 .

[29]  Jeffrey D. Karpicke,et al.  Separate mnemonic effects of retrieval practice and elaborative encoding , 2012 .

[30]  Ismael Pascual-Nieto,et al.  Computer-assisted assessment of free-text answers , 2009, The Knowledge Engineering Review.

[31]  Henry L. Roediger,et al.  Repeated retrieval during learning is the key to long-term retention , 2007 .

[32]  M. McDaniel,et al.  Test-enhanced learning in the classroom: long-term improvements from quizzing. , 2011, Journal of experimental psychology. Applied.

[33]  Katherine A Rawson,et al.  How and when do students use flashcards? , 2012, Memory.

[34]  Peter W. Foltz,et al.  The intelligent essay assessor: Applications to educational technology , 1999 .

[35]  Mark A. McDaniel,et al.  Test-Enhanced Learning in a Middle School Science Classroom: The Effects of Quiz Frequency and Placement. , 2011 .

[36]  J. Fleiss,et al.  Statistical methods for rates and proportions , 1973 .

[37]  Martin Chodorow,et al.  C-rater: Automated Scoring of Short-Answer Questions , 2003, Comput. Humanit..

[38]  Katherine A. Rawson,et al.  The interim test effect: Testing prior material can facilitate the learning of new material , 2011, Psychonomic bulletin & review.

[39]  Rada Mihalcea,et al.  Learning to Grade Short Answer Questions using Semantic Similarity Measures and Dependency Graph Alignments , 2011, ACL.

[40]  Rada Mihalcea,et al.  Text-to-Text Semantic Similarity for Automatic Short Answer Grading , 2009, EACL.

[41]  M. McDaniel,et al.  The Read-Recite-Review Study Strategy , 2009, Psychological science.

[42]  A. C. Butler,et al.  The critical role of retrieval practice in long-term retention , 2011, Trends in Cognitive Sciences.

[43]  Jeffrey D. Karpicke,et al.  Metacognitive control and strategy selection: deciding to practice retrieval during learning. , 2009, Journal of experimental psychology. General.

[44]  Fakhroddin Noorbehbahani,et al.  The automatic assessment of free text answers using a modified BLEU algorithm , 2011, Comput. Educ..

[45]  A. Scott,et al.  A simple method for the analysis of clustered binary data. , 1992, Biometrics.

[46]  Gerry Stahl,et al.  Developing Summarization Skills through the Use of LSA-Based Feedback , 2000, Interact. Learn. Environ..

[47]  Andrew C Butler,et al.  Repeated testing produces superior transfer of learning relative to repeated studying. , 2010, Journal of experimental psychology. Learning, memory, and cognition.

[48]  Robert A. Bjork,et al.  The promise and perils of self-regulated study , 2007, Psychonomic bulletin & review.

[49]  Stephen Pulman,et al.  Auto−marking 2: An update on the UCLES−Oxford University research into using computational linguistics to score short‚ free text responses , 2004 .

[50]  John Dunlosky,et al.  Optimizing schedules of retrieval practice for durable and efficient learning: how much is enough? , 2011, Journal of experimental psychology. General.