Predicting Robust Learning With the Visual Form of the Moment-by-Moment Learning Curve

We present a new method for analyzing a student's learning over time for a specific skill: analysis of the graph of the student's moment-by-moment learning over time. Moment-by-moment learning is calculated using a data-mined model that assesses the probability that a student learned a skill or concept at a specific time during learning (Baker, Goldstein, & Heffernan, 2010, 2011). Two coders labeled data from students who used an intelligent tutoring system for college genetics. They coded in terms of 7 forms that the moment-by-moment learning curve can take. These labels are correlated to test data on the robustness of students’ learning. We find that different visual forms are correlated with very different learning outcomes. This work suggests that analysis of moment-by-moment learning curves may be able to shed light on the implications of students’ different patterns of learning over time.

[1]  Ingo Mierswa,et al.  YALE: rapid prototyping for complex data mining tasks , 2006, KDD '06.

[2]  R. Nisbett,et al.  Immediate and delayed transfer of training effects in statistical reasoning. , 1991, Journal of experimental psychology. General.

[3]  Ryan Shaun Joazeiro de Baker,et al.  Leveraging machine-learned detectors of systematic inquiry behavior to estimate and predict transfer of inquiry skill , 2011, User Modeling and User-Adapted Interaction.

[4]  Ryan Shaun Joazeiro de Baker,et al.  Modeling and understanding students' off-task behavior in intelligent tutoring systems , 2007, CHI.

[5]  G. Bower,et al.  REVERSALS PRIOR TO SOLUTION IN CONCEPT IDENTIFICATION. , 1963, Journal of experimental psychology.

[6]  Kenneth R. Koedinger,et al.  Is Over Practice Necessary? - Improving Learning Efficiency with the Cognitive Tutor through Educational Data Mining , 2007, AIED.

[7]  Kenneth R. Koedinger,et al.  Learning Factors Analysis - A General Method for Cognitive Model Evaluation and Improvement , 2006, Intelligent Tutoring Systems.

[8]  Antonija Mitrovic,et al.  Evaluating and improving adaptive educational systems with learning curves , 2011, User Modeling and User-Adapted Interaction.

[9]  Edward M. Bowden,et al.  New approaches to demystifying insight , 2005, Trends in Cognitive Sciences.

[10]  Albert T. Corbett,et al.  The Knowledge-Learning-Instruction (KLI) Framework: Toward Bridging the Science-Practice Chasm to Enhance Robust Student Learning , 2010 .

[11]  Allen and Rosenbloom Paul S. Newell,et al.  Mechanisms of Skill Acquisition and the Law of Practice , 1993 .

[12]  Scott D. Brown,et al.  The power law repealed: The case for an exponential law of practice , 2000, Psychonomic bulletin & review.

[13]  Ryan Shaun Joazeiro de Baker,et al.  Contextual Slip and Prediction of Student Performance after Use of an Intelligent Tutor , 2010, UMAP.

[14]  Sebastián Ventura,et al.  Educational data mining: A survey from 1995 to 2005 , 2007, Expert Syst. Appl..

[15]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[16]  D. Kuhn,et al.  Arguing on the computer: a microgenetic study of developing argument skills in a computer-supported environment. , 2008, Child development.

[17]  T. Perneger.  What's wrong with Bonferroni adjustments , 1998, BMJ.

[18]  Mark A. Hall,et al.  Correlation-based Feature Selection for Discrete and Numeric Class Machine Learning , 1999, ICML.

[19]  Albert T. Corbett,et al.  The Knowledge-Learning-Instruction Framework: Bridging the Science-Practice Chasm to Enhance Robust Student Learning , 2012, Cogn. Sci..

[20]  John R. Anderson.  Cognitive Skills and Their Acquisition , 2013 .

[21]  Daniel L. Schwartz,et al.  Rethinking transfer: A simple proposal with multiple implica-tions , 1999 .

[22]  Gordon H. Bower,et al.  Application of a model to paired-associate learning , 1961 .

[23]  Vincent Aleven,et al.  More Accurate Student Modeling through Contextual Estimation of Slip and Guess Probabilities in Bayesian Knowledge Tracing , 2008, Intelligent Tutoring Systems.

[24]  Arnon Hershkovitz,et al.  Learning about Online Learning Processes and Students' Motivation through Web Usage Mining , 2009 .

[25]  Albert T. Corbett,et al.  A Cognitive Tutor for Genetics Problem Solving: Learning Gains and Student Modeling , 2010 .

[26]  Sidney D'Mello,et al.  The half-life of cognitive-affective states during complex learning , 2011, Cognition & emotion.

[27]  John R. Anderson,et al.  The Transfer of Cognitive Skill , 1989 .

[28]  Paul Cobb,et al.  Individual and Collective Mathematical Development: The Case of Statistical Data Analysis. , 1999 .

[29]  W. Estes.  Toward a Statistical Theory of Learning. , 1994 .

[30]  Richard B. Anderson,et al.  Artifactual power curves in forgetting , 1997, Memory & cognition.

[31]  J. E. Mazur,et al.  Learning as accumulation: a reexamination of the learning curve. , 1978, Psychological bulletin.

[32]  Ryan Shaun Joazeiro de Baker,et al.  Automatically Detecting a Student's Preparation for Future Learning: Help Use is Key , 2011, EDM.

[33]  John D. Storey,et al.  Strong control, conservative point estimation and simultaneous conservative consistency of false discovery rates: a unified approach , 2004 .

[34]  Agneta Gulz,et al.  Catching Eureka on the Fly , 2008, AAAI Spring Symposium: Emotion, Personality, and Social Behavior.

[35]  Zachary A. Pardos,et al.  Ensembling predictions of student knowledge within intelligent tutoring systems , 2011, UMAP'11.

[36]  Daniel L. Schwartz,et al.  Chapter 3: Rethinking Transfer: A Simple Proposal With Multiple Implications , 1999 .

[37]  Jacob Cohen.  A Coefficient of Agreement for Nominal Scales , 1960 .

[38]  John R. Anderson,et al.  Knowledge tracing: Modeling the acquisition of procedural knowledge , 2005, User Modeling and User-Adapted Interaction.

[39]  Ryan Shaun Joazeiro de Baker,et al.  Developing a generalizable detector of when students game the system , 2008, User Modeling and User-Adapted Interaction.

[40]  Neil T. Heffernan,et al.  Detecting Learning Moment-by-Moment , 2011, Int. J. Artif. Intell. Educ..

[41]  Kurt VanLehn,et al.  Andes: A Coached Problem Solving Environment for Physics , 2000, Intelligent Tutoring Systems.

[42]  Eric A. Jenkins,et al.  How Children Discover New Strategies , 1989 .

[43]  Ryan Shaun Joazeiro de Baker,et al.  Using Text Replay Tagging to Produce Detectors of Systematic Experimentation Behavior Patterns , 2010, EDM.

[44]  Y. Benjamini,et al.  Controlling the false discovery rate: a practical and powerful approach to multiple testing , 1995 .

[45]  Albert T. Corbett,et al.  Human Classification of Low-Fidelity Replays of Student Actions , 2006 .

[46]  R. Schmidt,et al.  New Conceptualizations of Practice: Common Principles in Three Paradigms Suggest New Concepts for Training , 1992 .

[47]  Jack Mostow,et al.  Evaluating tutors that listen: an overview of project LISTEN , 2001 .

[48]  Richard Lehrer,et al.  Similarity of form and substance: modeling material kind , 2001 .

[49]  Neil T. Heffernan,et al.  Detecting the Moment of Learning , 2010, Intelligent Tutoring Systems.

[50]  L. Schauble,et al.  Cross-Domain Development of Scientific Reasoning , 1992 .

[51]  J. R. Landis,et al.  The measurement of observer agreement for categorical data. , 1977, Biometrics.

[52]  Kathryn Henderson,et al.  On Line and On Paper: Visual Representations, Visual Culture, and Computer Graphics in Design Engineering , 1998 .

[53]  John R. Anderson,et al.  Practice and Forgetting Effects on Vocabulary Memory: An Activation-Based Model of the Spacing Effect , 2005, Cogn. Sci..

[54]  K. Koedinger,et al.  Fostering the Intelligent Novice: Learning From Errors With Metacognitive Tutoring , 2005 .

[55]  Ryan S. Baker,et al.  The State of Educational Data Mining in 2009: A Review and Future Visions. , 2009, EDM 2009.

[56]  Joseph E. Beck,et al.  Using learning decomposition to analyze student fluency development , 2005 .

[57]  Paul J. Feltovich,et al.  Smart machines in education: the coming revolution in educational technology , 2001 .

[58]  K. Crowley,et al.  The microgenetic method. A direct means for studying cognitive development. , 1991, The American psychologist.

[59]  Jack Mostow,et al.  How Who Should Practice: Using Learning Decomposition to Evaluate the Efficacy of Different Types of Practice for Different Types of Students , 2008, Intelligent Tutoring Systems.