Towards Understanding Expert Coding of Student Disengagement in Online Learning

Gaming the system, a behavior where students disengage from a learning environment and attempt to succeed by exploiting properties of the system, has been shown to be associated with lower learning. Machine learned and knowledge engineered models have been created to identify gaming behaviors, but few efforts have been made to precisely identify how experts code gaming behaviors. In this paper, we used cognitive task analysis to elicit knowledge about how experts code students as gaming or not in Cognitive Tutor Algebra. We show how building a cognitive model of this process gave us insights about the behaviors gaming is composed of.

[1]  Ryan Shaun Joazeiro de Baker,et al.  Labeling Student Behavior Faster and More Precisely with Text Replays , 2008, EDM.

[2]  David F. Feldon,et al.  Cognitive task analysis , 2009 .

[3]  Nancy J. Cooke,et al.  Varieties of knowledge elicitation techniques , 1994, Int. J. Hum. Comput. Stud..

[4]  Albert T. Corbett,et al.  Human Classification of Low-Fidelity Replays of Student Actions , 2006 .

[5]  Stephen E. Fancsali.  Data-driven causal modeling of “ gaming the system ” and off-task behavior in Cognitive Tutor Algebra , 2013 .

[6]  Mary A. Meyer,et al.  How to apply the anthropological technique of participant observation to knowledge acquisition for expert systems , 1992, IEEE Trans. Syst. Man Cybern..

[7]  Neil T. Heffernan,et al.  Detection and Analysis of Off-Task Gaming Behavior in Intelligent Tutoring Systems , 2006, Intelligent Tutoring Systems.

[8]  M. W. van Someren,et al.  The think aloud method: a practical approach to modelling cognitive processes , 1994 .

[9]  Neil T. Heffernan,et al.  The Fine-Grained Impact of Gaming (?) on Learning , 2010, Intelligent Tutoring Systems.

[10]  Ryan Shaun Joazeiro de Baker,et al.  Developing a generalizable detector of when students game the system , 2008, User Modeling and User-Adapted Interaction.

[11]  Thomas Bay,et al.  Gaming the system , 2011 .

[12]  Beverly Park Woolf,et al.  Repairing Disengagement With Non-Invasive Interventions , 2007, AIED.

[13]  Kasia Muldner,et al.  An analysis of students’ gaming behaviors in an intelligent tutoring system: predictors and impacts , 2011, User Modeling and User-Adapted Interaction.

[14]  Neil T. Heffernan,et al.  Prevention of Off-Task Gaming Behavior in Intelligent Tutoring Systems , 2006, Intelligent Tutoring Systems.

[15]  Lei Qu,et al.  Classifying Learner Engagement through Integration of Multiple Data Sources , 2006, AAAI.

[16]  Arnon Hershkovitz,et al.  The Impact of Off-task and Gaming Behaviors on Learning: Immediate or Aggregate? , 2009, AIED.

[17]  Jacob Cohen.  A Coefficient of Agreement for Nominal Scales , 1960 .

[18]  Vladimir I. Levenshtein,et al.  Binary codes capable of correcting deletions, insertions, and reversals , 1965 .

[19]  Beverly Park Woolf,et al.  A Dynamic Mixture Model to Detect Student Motivation and Proficiency , 2006, AAAI.

[20]  Peter Maurer,et al.  The Cambridge Handbook of the Learning Sciences , 2022 .

[21]  Ryan Shaun Joazeiro de Baker,et al.  Towards Predicting Future Transfer of Learning , 2011, AIED.

[22]  A. Corbett,et al.  The Cambridge Handbook of the Learning Sciences: Cognitive Tutors , 2005 .

[23]  Moffat Mathews,et al.  Detecting Gaming the System in Constraint-Based Tutors , 2010, UMAP.

[24]  J. Michael Spector,et al.  Handbook of Research on Educational Communications and Technology, 3rd Edition , 2012 .

[25]  Zachary A. Pardos,et al.  Affective states and state tests: investigating how affect throughout the school year predicts end of year learning outcomes , 2013, LAK '13.

[26]  Arthur C. Graesser,et al.  Better to be frustrated than bored: The incidence, persistence, and impact of learners' cognitive-affective states during interactions with three different computer-based learning environments , 2010, Int. J. Hum. Comput. Stud..

[27]  Vincent Aleven,et al.  Toward Meta-cognitive Tutoring: A Model of Help Seeking with a Cognitive Tutor , 2006, Int. J. Artif. Intell. Educ..

[28]  Neil T. Heffernan,et al.  Predicting College Enrollment from Student Interaction with an Intelligent Tutoring System in Middle School , 2013, EDM.