Data-driven causal modeling of “ gaming the system ” and off-task behavior in Cognitive Tutor Algebra

“Gaming the system” and off-task behavior in intelligent tutoring systems (ITSs) have been found to be negatively associated with student learning outcomes. We summarize recent work to determine whether these behaviors are likely causes of decreased learning in the Cognitive Tutor R © Algebra ITS using algorithmic search for the structure of graphical causal models. We apply data-driven, software “detectors” of these behaviors to observed log data for 102 adult learners in an algebra course at the University of Phoenix R ©. We find evidence that “gaming the system” is a cause of decreased learning and that, while the two are correlated, off-task behavior is not a cause of decreased learning. A posited measure of the extent to which students manifest “known misconceptions” in the tutor mediates the causal link between gaming the system and learning. We discuss these results and several future research topics.

[1]  Richard Scheines,et al.  Replacing Lecture with Web-Based Course Materials , 2005 .

[2]  Ryan Shaun Joazeiro de Baker,et al.  Developing a generalizable detector of when students game the system , 2008, User Modeling and User-Adapted Interaction.

[3]  Albert T. Corbett,et al.  Cognitive Tutor: Applied research in mathematics education , 2007, Psychonomic bulletin & review.

[4]  Vincent Aleven,et al.  Does Representational Understanding Enhance Fluency - Or Vice Versa? Searching for Mediation Models , 2013, EDM.

[5]  R. P. McDonald,et al.  Structural Equations with Latent Variables , 1989 .

[6]  Tom Burr,et al.  Causation, Prediction, and Search , 2003, Technometrics.

[7]  Stephen Fancsali.  Variable Construction and Causal Discovery for Cognitive Tutor Log Data: Initial Results , 2012, EDM.

[8]  Vincent Aleven,et al.  Limitations of Student Control: Do Students Know When They Need Help? , 2000, Intelligent Tutoring Systems.

[9]  E. Maris.  Psychometric latent response models , 1995 .

[10]  Ryan Shaun Joazeiro de Baker,et al.  Detecting Student Misuse of Intelligent Tutoring Systems , 2004, Intelligent Tutoring Systems.

[11]  John R. Anderson,et al.  Knowledge tracing: Modeling the acquisition of procedural knowledge , 2005, User Modeling and User-Adapted Interaction.

[12]  Kenneth R. Koedinger,et al.  A Response Time Model For Bottom-Out Hints as Worked Examples , 2008, EDM.

[13]  P. Spirtes,et al.  Causation, Prediction, and Search, 2nd Edition , 2001 .

[14]  Joseph E. Beck,et al.  Engagement tracing: using response times to model student disengagement , 2005, AIED.

[15]  Vincent Aleven,et al.  Sensor-free automated detection of affect in a Cognitive Tutor for Algebra , 2012, EDM.

[16]  Thomas S. Richardson,et al.  A Discovery Algorithm for Directed Cyclic Graphs , 1996, UAI.

[17]  J. Pearl.  Causality: Models, Reasoning and Inference , 2000 .

[18]  Neil T. Heffernan,et al.  Prevention of Off-Task Gaming Behavior in Intelligent Tutoring Systems , 2006, Intelligent Tutoring Systems.

[19]  Lei Qu,et al.  Classifying Learner Engagement through Integration of Multiple Data Sources , 2006, AAAI.

[20]  Ryan Shaun Joazeiro de Baker,et al.  Modeling and understanding students' off-task behavior in intelligent tutoring systems , 2007, CHI.

[21]  Joseph E. Beck,et al.  Exploring User Data From a Game-like Math Tutor: A Case Study in Causal Modeling , 2011, EDM.

[22]  Ryan Shaun Joazeiro de Baker,et al.  Labeling Student Behavior Faster and More Precisely with Text Replays , 2008, EDM.

[23]  Stephen Fancsali,et al.  Constructing Variables Th at Support Causal Inference , 2013 .

[24]  Ryan Shaun Joazeiro de Baker,et al.  Adapting to When Students Game an Intelligent Tutoring System , 2006, Intelligent Tutoring Systems.

[25]  Arnon Hershkovitz,et al.  The Impact of Off-task and Gaming Behaviors on Learning: Immediate or Aggregate? , 2009, AIED.