Using Response Times to Model Not-Reached Items due to Time Limits

Missing values at the end of a test typically are the result of test takers running out of time and can as such be understood by studying test takers’ working speed. As testing moves to computer-based assessment, response times become available allowing to simulatenously model speed and ability. Integrating research on response time modeling with research on modeling missing responses, we propose using response times to model missing values due to time limits. We identify similarities between approaches used to account for not-reached items (Rose et al. in ETS Res Rep Ser 2010:i–53, 2010) and the speed-accuracy (SA) model for joint modeling of effective speed and effective ability as proposed by van der Linden (Psychometrika 72(3):287–308, 2007). In a simulation, we show (a) that the SA model can recover parameters in the presence of missing values due to time limits and (b) that the response time model, using item-level timing information rather than a count of not-reached items, results in person parameter estimates that differ from missing data IRT models applied to not-reached items. We propose using the SA model to model the missing data process and to use both, ability and speed, to describe the performance of test takers. We illustrate the application of the model in an empirical analysis.

[1]  Kentaro Yamamoto,et al.  ESTIMATING THE EFFECTS OF TEST LENGTH AND TEST TIME ON PARAMETER ESTIMATION USING THE HYBRID MODEL , 1995 .

[2]  Barbara S. Plake,et al.  The Impact of Omitted Responses on the Accuracy of Ability Estimation in Item Response Theory , 2001 .

[3]  Jonathan P. Weeks,et al.  Using Response Time Data to Inform the Coding of Omitted Responses , 2016 .

[4]  M. Davier,et al.  Modeling Nonignorable Missing Data with Item Response Theory (IRT). Research Report. ETS RR-10-11. , 2010 .

[5]  Holmes Finch,et al.  Estimation of Item Response Theory Parameters in the Presence of Missing Data , 2008 .

[6]  R. H. Klein Entink,et al.  A Multivariate Multilevel Approach to the Modeling of Accuracy and Speed of Test Takers , 2008, Psychometrika.

[7]  Yi-Hsuan Lee,et al.  Using response time to investigate students' test-taking behaviors in a NAEP computer-based study , 2014, Large-scale Assessments in Education.

[8]  Hua-Hua Chang,et al.  A Conditional Joint Modeling Approach for Locally Dependent Item Responses and Response Times , 2015 .

[9]  Karoline A. Sachse,et al.  When Nonresponse Mechanisms Change: Effects on Trends and Group Comparisons in International Large-Scale Assessments , 2019, Educational and psychological measurement.

[10]  Claus H. Carstensen,et al.  Investigating Mechanisms for Missing Responses in Competence Tests , 2015 .

[11]  Willem J. van der Linden,et al.  A lognormal model for response times on test items , 2006 .

[12]  Steffi Pohl,et al.  Dealing With Item Nonresponse in Large‐Scale Cognitive Assessments: The Impact of Missing Data Methods on Estimated Explanatory Relationships , 2017 .

[13]  Wim J. van der Linden,et al.  Bayesian Procedures for Identifying Aberrant Response-Time Patterns in Adaptive Testing , 2008 .

[14]  Allan S. Cohen,et al.  A Speeded Item Response Model with Gradual Process Change , 2008 .

[15]  Jochen Ranger,et al.  The case of dependency of responses and response times: A modeling approach based on standard latent trait models , 2012 .

[16]  Claus H. Carstensen,et al.  Taking the Missing Propensity Into Account When Estimating Competence Scores , 2015, Educational and psychological measurement.

[17]  Rebecca Holman,et al.  Modelling non-ignorable missing-data mechanisms with item response theory models. , 2005, The British journal of mathematical and statistical psychology.

[18]  Frank Goldhammer,et al.  Controlling Individuals’ Time Spent on Task in Speeded Performance Measures , 2014 .

[19]  Nancy L. Allen,et al.  The NAEP 1998 Technical Report. , 2001 .

[20]  Frank Goldhammer,et al.  Measuring Ability, Speed, or Both? Challenges, Psychometric Solutions, and What Can Be Gained From Experimental Control , 2015, Measurement : interdisciplinary research and perspectives.

[21]  Changes in achievement on PISA: the case of Ireland and implications for international assessment practice , 2013 .

[22]  Frederic M. Lord,et al.  Estimation of latent ability and item parameters when there are omitted responses , 1974 .

[23]  P. Schönemann,et al.  Power Tables for Analysis of Variance , 1978 .

[24]  Eugene G. Johnson.  The NAEP 1990 Technical Report. , 1992 .

[25]  Andrew Gelman,et al.  Inference from Simulations and Monitoring Convergence , 2011 .

[26]  P. Boeck,et al.  Modelling Conditional Dependence Between Response Time and Accuracy , 2017, Psychometrika.

[27]  Martyn Plummer,et al.  JAGS: A program for analysis of Bayesian graphical models using Gibbs sampling , 2003 .

[28]  Deborah L. Schnipke,et al.  Modeling Item Response Times With a Two-State Mixture Model: A New Method of Measuring , 1997 .

[29]  Steffi Pohl,et al.  Dealing With Omitted and Not-Reached Items in Competence Tests , 2014 .

[30]  Jochen Ranger,et al.  Measuring Speed, Ability, or Motivation: A Comment on Goldhammer (2015) , 2015 .

[31]  Francis Tuerlinckx,et al.  A generalized linear factor model approach to the hierarchical framework for responses and response times. , 2015, The British journal of mathematical and statistical psychology.

[32]  John K. Kruschke,et al.  Doing Bayesian Data Analysis: A Tutorial with R, JAGS, and Stan , 2014 .

[33]  Jochen Ranger,et al.  A flexible latent trait model for response times in tests , 2012 .

[34]  Maria Bolsinova,et al.  Response moderation models for conditional dependence between response time and response accuracy , 2017, The British journal of mathematical and statistical psychology.

[35]  Daniel Oberski,et al.  Hidden Markov Item Response Theory Models for Responses and Response Times , 2016, Multivariate behavioral research.

[36]  van der Linden,et al.  A hierarchical framework for modeling speed and accuracy on test items , 2007 .

[37]  Krista Breithaupt,et al.  Detecting Differential Speededness in Multistage Testing , 2007 .

[38]  Alexei J Drummond,et al.  Estimating mutation parameters, population history and genealogy simultaneously from temporally spaced sequence data. , 2002, Genetics.

[39]  J. Fox.  Bayesian Item Response Modeling: Theory and Applications , 2010 .

[40]  Commentary: On the Importance of the Speed-Ability Trade-Off When Dealing With Not Reached Items , 2018, Front. Psychol..

[41]  Wim J. van der Linden,et al.  Using Response-Time Constraints to Control for Differential Speededness in Computerized Adaptive Testing , 1999 .

[42]  D. Rubin,et al.  Inference from Iterative Simulation Using Multiple Sequences , 1992 .

[43]  Jean-Paul Fox,et al.  Bayesian Item Response Modeling , 2010 .

[44]  Martin Knott,et al.  Weighting for item non‐response in attitude scales by using latent variable models with covariates , 2000 .

[45]  Willem J. van der Linden,et al.  Using Response Times for Item Selection in Adaptive Testing , 2008 .

[46]  M. Bolsinova,et al.  On the Importance of the Speed-Ability Trade-Off When Dealing With Not Reached Items , 2018, Front. Psychol..

[47]  Yi-Hsuan Lee,et al.  A review of recent response-time analyses in educational testing , 2011 .

[48]  Pao-Kuei Wu,et al.  MISSING RESPONSES AND IRT ABILITY ESTIMATION: OMITS, CHOICE, TIME LIMITS, AND ADAPTIVE TESTING , 1996 .

[49]  J. Fox,et al.  Joint Modeling of Ability and Differential Speed Using Responses and Response Times , 2016, Multivariate behavioral research.

[50]  Sun Hee Kim,et al.  An item response theory approach to longitudinal analysis with application to summer setback in preschool language/literacy , 2013 .

[51]  C. Glas,et al.  Nonignorable data in IRT models: Polytomous responses and response propensity models with covariates , 2015 .

[52]  M. Davison,et al.  Modeling Individual Differences in Numerical Reasoning Speed as a Random Effect of Response Time Limits , 2011 .

[53]  Colm O'Muircheartaigh,et al.  Symmetric pattern models: a latent variable approach to item non‐response in attitude scales , 1999 .

[54]  Christine E. DeMars,et al.  Low Examinee Effort in Low-Stakes Assessment: Problems and Potential Solutions , 2005 .

[55]  Cornelis A.W. Glas,et al.  Statistical Tests of Conditional Independence Between Responses and/or Response Times on Test Items , 2010 .