Detecting Gaming the System in Constraint-Based Tutors

Recently, detectors of gaming the system have been developed for several intelligent tutoring systems where the problem-solving process is reified, and gaming consists of systematic guessing and help abuse Constraint-based tutors differ from the tutors where gaming detectors have previously been developed on several dimensions: in particular, higher-level answers are assessed according to a larger number of finer-grained constraints, and feedback is split into levels rather than an entire help sequence being available at any time Correspondingly, help abuse behaviors differ, including behaviors such as rapidly repeating the same answer or blank answers to elicit answers We use text replay labeling in combination with educational data mining methods to create a gaming detector for SQL-Tutor, a popular constraint-based tutor This detector assesses gaming at the level of multiple-submission sequences and is accurate both at identifying gaming within submission sequences and at identifying how much each student games the system It achieves only limited success, however, at distinguishing different types of gaming behavior from each other.

[1]  Ryan Shaun Joazeiro de Baker,et al.  Adapting to When Students Game an Intelligent Tutoring System , 2006, Intelligent Tutoring Systems.

[2]  Beverly Park Woolf,et al.  A Dynamic Mixture Model to Detect Student Motivation and Proficiency , 2006, AAAI.

[3]  Ryan Shaun Joazeiro de Baker,et al.  Affect and Usage Choices in Simulation Problem-Solving Environments , 2007, AIED.

[4]  Antonija Mitrovic,et al.  An Intelligent SQL Tutor on the Web , 2003, Int. J. Artif. Intell. Educ..

[5]  Moffat Mathews,et al.  How Does Students' Help-Seeking Behaviour Affect Learning? , 2008, Intelligent Tutoring Systems.

[6]  Antonija Mitrovic,et al.  Evaluating Adaptive Problem Selection , 2004, AH.

[7]  Kurt VanLehn,et al.  Effects of Dissuading Unnecessary Help Requests While Providing Proactive Help , 2005, AIED.

[8]  Albert T. Corbett,et al.  Human Classification of Low-Fidelity Replays of Student Actions , 2006 .

[9]  Kenneth R. Koedinger,et al.  A Response Time Model For Bottom-Out Hints as Worked Examples , 2008, EDM.

[10]  Richard C. Anderson,et al.  FEEDBACK PROCEDURES IN COMPUTER-ASSISTED ARITHMETIC INSTRUCTION , 1973 .

[11]  Ryan Shaun Joazeiro de Baker,et al.  Off-task behavior in the cognitive tutor classroom: when students "game the system" , 2004, CHI.

[12]  Ingo Mierswa,et al.  YALE: rapid prototyping for complex data mining tasks , 2006, KDD '06.

[13]  Joseph E. Beck,et al.  Engagement tracing: using response times to model student disengagement , 2005, AIED.

[14]  John R. Anderson,et al.  Knowledge tracing: Modeling the acquisition of procedural knowledge , 2005, User Modeling and User-Adapted Interaction.

[15]  Peter Gobel.  Student Off-task Behavior and Motivation in the CALL Classroom , 2008 .

[16]  Vincent Aleven,et al.  Toward Meta-cognitive Tutoring: A Model of Help Seeking with a Cognitive Tutor , 2006, Int. J. Artif. Intell. Educ..

[17]  Christos Faloutsos,et al.  Density biased sampling: an improved method for data mining and clustering , 2000, SIGMOD '00.

[18]  Julita Vassileva,et al.  Design and evaluation of an adaptive incentive mechanism for sustained educational online communities , 2006, User Modeling and User-Adapted Interaction.

[19]  Vincent Aleven,et al.  Educational Software Features that Encourage and Discourage "Gaming the System" , 2009, AIED.

[20]  Michael Heilman,et al.  Language Learning: Challenges for Intelligent Tutoring Systems , 2006 .

[21]  J. Hanley,et al.  The meaning and use of the area under a receiver operating characteristic (ROC) curve. , 1982, Radiology.

[22]  Lei Qu,et al.  Mathematics motivation and achievement as predictors of high school students' guessing and help-seeking with instructional software , 2008, J. Comput. Assist. Learn..

[23]  D. Wood,et al.  Help seeking, learning and contingent tutoring , 1999, Comput. Educ..

[24]  Ryan Shaun Joazeiro de Baker,et al.  Developing a generalizable detector of when students game the system , 2008, User Modeling and User-Adapted Interaction.

[25]  Neil T. Heffernan,et al.  Detection and Analysis of Off-Task Gaming Behavior in Intelligent Tutoring Systems , 2006, Intelligent Tutoring Systems.

[26]  Kurt VanLehn,et al.  The Behavior of Tutoring Systems , 2006, Int. J. Artif. Intell. Educ..

[27]  Carlo Strapparava,et al.  Adaptive Hypermedia and Adaptive Web-Based Systems, 5th International Conference, AH 2008, Hannover, Germany, July 29 - August 1, 2008. Proceedings , 2008, AH.

[28]  Ryan Shaun Joazeiro de Baker,et al.  Labeling Student Behavior Faster and More Precisely with Text Replays , 2008, EDM.

[29]  Thomas Bay,et al.  Gaming the system , 2011 .

[30]  J. Schofield.  Computers and classroom culture , 1995 .

[31]  Beverly Park Woolf,et al.  Repairing Disengagement With Non-Invasive Interventions , 2007, AIED.