Comparing of Traditional Assessment with Dynamic Testing in a Tutoring System

It would be great if ITS can also be used to do the benchmark assessment, so that no time from instruction is “stolen” to do extra assessments. It is presumed that given a limited amount of time for assessing, people should give a test but not wasting time giving students feedback. However, Feng and Heffernan (2010) compared two simulated conditions and found that the condition that lets students get feedback during a test was actually superior (not statistically reliable) at predicting student performance than the “test” condition, in which students did about double the number of problems. In this study, we address the weakness in Feng & Heffernan (2010) (i.e. simulated conditions) and run a new randomized control trial in a tutoring system with participants from two different grades, 7 and 8 to see if the main effect would replicate. Our results suggest that unlike our previous results 1) there is no reliable main effect across all students; 2) the dynamic testing condition works better with 7 graders than with 8 graders. We also find that 7 graders and 8 graders behaved differently while working within the system, which appeared to be related to “gaming”.

[1]  Neil T. Heffernan,et al.  Addressing the assessment challenge with an online system that tutors as it assesses , 2009, User Modeling and User-Adapted Interaction.

[2]  Ryan Shaun Joazeiro de Baker,et al.  Off-task behavior in the cognitive tutor classroom: when students "game the system" , 2004, CHI.

[3]  T. Schack,et al.  Dynamic testing , 2003 .

[4]  Peter Tiño,et al.  Managing Diversity in Regression Ensembles , 2005, J. Mach. Learn. Res..

[5]  Sarah Guckenburg,et al.  Measuring how benchmark assessments affect student achievement , 2007 .

[6]  Ana I. González Acuña.  An experimental comparison of three methods for constructing ensembles of decision trees: Bagging, Boosting, and Randomization , 2012 .

[7]  A. Onwuegbuzie,et al.  The Effect of Time Constraints and Statistics Test Anxiety on Test Performance in a Statistics Course , 1995 .

[8]  Zachary A. Pardos,et al.  Clustering Students to Generate an Ensemble to Improve Standard Test Score Predictions , 2011, AIED.

[9]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[10]  L. Fuchs,et al.  Dynamic Assessment of Algebraic Learning in Predicting Third Graders' Development of Mathematical Problem Solving. , 2008, Journal of educational psychology.

[11]  Neil T. Heffernan,et al.  Can We Get Better Assessment From A Tutoring System Compared to Traditional Paper Testing? Can We Have Our Cake (Better Assessment) and Eat It too (Student Learning During the Test)? , 2010, EDM.

[12]  Ann L. Brown,et al.  Dynamic Assessment: One Approach and Some Initial Data. Technical Report No. 361. , 1985 .

[13]  Lynn S. Fuchs,et al.  Dynamic Assessment as Responsiveness to Intervention; a Scripted Protocol to Identify Young At-Risk Readers , 2007 .

[14]  Thomas G. Dietterich.  Multiple Classifier Systems , 2000, Lecture Notes in Computer Science.