A New Procedure for Detection of Students’ Rapid Guessing Responses Using Response Time

ABSTRACT Unmotivated test takers using rapid guessing in item responses can affect validity studies and teacher and institution performance evaluation negatively, making it critical to identify these test takers. The authors propose a new nonparametric method for finding response-time thresholds for flagging item responses that result from rapid-guessing behavior. Using data from a low-stakes assessment of college-level academic skills as an illustration, the authors evaluate and compare model fit and score estimation based on data sets cleaned by both new and existing methods. Flagging rapid-guessing responses is found to generally improve model fit, item parameter estimation, and score estimation, as in the literature. This new method, based on both response time and response accuracy, shows promise in detecting rapid guessing and in improving efficiency of the flagging process when built into data analysis.

[1]  Wim J. van der Linden,et al.  Bayesian Procedures for Identifying Aberrant Response-Time Patterns in Adaptive Testing , 2008 .

[2]  M. R. Novick,et al.  Statistical Theories of Mental Test Scores. , 1971 .

[3]  Rob R. Meijer,et al.  CUSUM-Based Person-Fit Statistics for Adaptive Testing , 2001 .

[4]  Steven L. Wise,et al.  Response Time Effort: A New Measure of Examinee Motivation in Computer-Based Tests , 2005 .

[5]  R. Meijer,et al.  Diagnosing item score patterns on a test using item response theory-based person-fit statistics. , 2003, Psychological methods.

[6]  Identifiers California,et al.  Annual Meeting of the National Council on Measurement in Education , 1998 .

[7]  van der Linden,et al.  A hierarchical framework for modeling speed and accuracy on test items , 2007 .

[8]  Shelby J. Haberman,et al.  A GENERAL PROGRAM FOR ITEM‐RESPONSE ANALYSIS THAT EMPLOYS THE STABILIZED NEWTON–RAPHSON ALGORITHM , 2013 .

[9]  Shelby J. Haberman,et al.  A Stabilized Newton-Raphson Algorithm for Log-Linear Models for Frequency Tables Derived by Indirect Observation , 1988 .

[10]  S. Wise,et al.  Setting Response Time Thresholds for a CAT Item Pool: The Normative Threshold Method , 2012 .

[11]  Steven L. Wise,et al.  An Application of Item Response Time: The Effort‐Moderated IRT Model , 2006 .

[12]  Shameem Nyla.  NATIONAL COUNCIL ON MEASUREMENT IN EDUCATION , 2004 .

[13]  Jorge N. Tendeiro,et al.  A CUSUM to Detect Person Misfit , 2012 .

[14]  J Christine Harmes,et al.  Two Approaches for Identifying Low-Motivated Students in a Low-Stakes Assessment Context , 2011 .

[15]  Wim J. Linden,et al.  Using response times to detect aberrant responses in computerized adaptive testing , 2003 .

[16]  Steven L. Wise,et al.  Setting the Response Time Threshold Parameter to Differentiate Solution Behavior From Rapid-Guessing Behavior , 2007 .

[17]  Shelby J. Haberman,et al.  Reporting of Subscores Using Multidimensional Item Response Theory , 2010 .

[18]  Yi-Hsuan Lee,et al.  A review of recent response-time analyses in educational testing , 2011 .

[19]  Steven L. Wise,et al.  An Investigation of the Differential Effort Received by Items on a Low-Stakes Computer-Based Test , 2006 .

[20]  Klaas Sijtsma,et al.  Methodology Review: Evaluating Person Fit , 2001 .

[21]  H. Gulliksen.  The relation of item difficulty and inter-item correlation to test variance and reliability , 1945 .

[22]  Rob R. Meijer,et al.  Person-Fit Research: An Introduction , 1996 .