Using Response Times for Joint Modeling of Response and Omission Behavior

Abstract For adequate modeling of missing responses, a thorough understanding of the nonresponse mechanisms is vital. As a large number of major testing programs are in the process or already have been moving to computer-based assessment, a rich body of additional data on examinee behavior becomes easily accessible. These additional data may contain valuable information on the processes associated with nonresponse. Bringing together research on item omissions with approaches for modeling response time data, we propose a framework for simultaneously modeling response behavior and omission behavior utilizing timing information for both. As such, the proposed model allows (a) to gain a deeper understanding of response and nonresponse behavior in general and, in particular, of the processes underlying item omissions in LSAs, (b) to model the processes determining the time examinees require to generate a response or to omit an item, and (c) to account for nonignorable item omissions. Parameter recovery of the proposed model is studied within a simulation study. An illustration of the model by means of an application to real data is provided.

[1]  Frederic M. Lord,et al.  Estimation of latent ability and item parameters when there are omitted responses , 1974 .

[2]  Frank Goldhammer,et al.  Measuring Ability, Speed, or Both? Challenges, Psychometric Solutions, and What Can Be Gained From Experimental Control , 2015, Measurement : interdisciplinary research and perspectives.

[3]  Steven L. Wise,et al.  Rapid-Guessing Behavior: Its Identification, Interpretation, and Implications , 2017 .

[4]  Frederic M. Lord,et al.  Maximum likelihood estimation of item response parameters when some responses are omitted , 1983 .

[5]  Wen-Chung Wang,et al.  A Multidimensional Hierarchical Framework for Modeling Speed and Ability in Computer-based Multidimensional Tests , 2018, 1807.04003.

[6]  R. H. Klein Entink,et al.  A Multivariate Multilevel Approach to the Modeling of Accuracy and Speed of Test Takers , 2008, Psychometrika.

[7]  Yi-Hsuan Lee,et al.  Using response time to investigate students' test-taking behaviors in a NAEP computer-based study , 2014, Large-scale Assessments in Education.

[8]  Claus H. Carstensen,et al.  Investigating Mechanisms for Missing Responses in Competence Tests , 2015 .

[9]  Steven L. Wise,et al.  Response Time Effort: A New Measure of Examinee Motivation in Computer-Based Tests , 2005 .

[10]  Benjamin Becker,et al.  Performance of missing data approaches under nonignorable missing data conditions , 2020, Methodology.

[11]  Joris Mulder,et al.  Prior Sensitivity Analysis in Default Bayesian Structural Equation Modeling , 2017, Psychological methods.

[12]  Norman Rose,et al.  Modeling Omitted and Not-Reached Items in IRT Models , 2017, Psychometrika.

[13]  Xiao-Li Meng,et al.  Modeling covariance matrices in terms of standard deviations and correlations, with application to shrinkage , 2000 .

[14]  D. Rubin,et al.  Inference from Iterative Simulation Using Multiple Sequences , 1992 .

[15]  Artur Pokropek,et al.  Grade of Membership Response Time Model for Detecting Guessing Behaviors , 2016 .

[16]  Claus H. Carstensen,et al.  Taking the Missing Propensity Into Account When Estimating Competence Scores , 2015, Educational and psychological measurement.

[17]  Andrew Gelman,et al.  The No-U-turn sampler: adaptively setting path lengths in Hamiltonian Monte Carlo , 2011, J. Mach. Learn. Res..

[18]  Matthias von Davier,et al.  A general diagnostic model applied to language testing data. , 2008, The British journal of mathematical and statistical psychology.

[19]  Dorota Kurowicka,et al.  Generating random correlation matrices based on vines and extended onion method , 2009, J. Multivar. Anal..

[20]  E L Hamaker,et al.  A Comparison of Inverse-Wishart Prior Specifications for Covariance Matrices in Multilevel Autoregressive Models , 2016, Multivariate behavioral research.

[21]  Chun Wang,et al.  A mixture hierarchical model for response times and response accuracy. , 2015, The British journal of mathematical and statistical psychology.

[22]  Wim J. van der Linden,et al.  IRT Parameter Estimation With Response Times as Collateral Information , 2010 .

[23]  Steffi Pohl,et al.  A Multiprocess Item Response Model for Not-Reached Items due to Time Limits and Quitting , 2019, Educational and psychological measurement.

[24]  Jean-Paul Fox,et al.  Bayesian Item Response Modeling , 2010 .

[25]  Jeffrey S. Rosenthal,et al.  Optimal Proposal Distributions and Adaptive MCMC , 2011 .

[26]  Federico Poloni Of Note , 2009 .

[27]  Russell G. Almond,et al.  DESIGN AND ANALYSIS IN A COGNITIVE ASSESSMENT , 2003 .

[28]  Francis Tuerlinckx,et al.  A generalized linear factor model approach to the hierarchical framework for responses and response times. , 2015, The British journal of mathematical and statistical psychology.

[29]  Joseph Hilbe,et al.  Data Analysis Using Regression and Multilevel/Hierarchical Models , 2009 .

[30]  Willem J. van der Linden,et al.  A lognormal model for response times on test items , 2006 .

[31]  G. Maris,et al.  Speed-Accuracy Response Models: Scoring Rules based on Response Time and Accuracy , 2012 .

[32]  Radford M. Neal MCMC Using Hamiltonian Dynamics , 2011, 1206.1901.

[33]  Steffi Pohl,et al.  Dealing With Item Nonresponse in Large‐Scale Cognitive Assessments: The Impact of Missing Data Methods on Estimated Explanatory Relationships , 2017 .

[34]  Benjamin Becker,et al.  Supplementary materials to: Performance of missing data approaches under nonignorable missing data conditions , 2020 .

[35]  J. Fox,et al.  A Box-Cox normal model for response times. , 2009, The British journal of mathematical and statistical psychology.

[36]  Mark D. Reckase,et al.  Using Response Time to Detect Item Preknowledge in Computer‐Based Licensure Examinations , 2016 .

[37]  J. Fox Bayesian Item Response Modeling: Theory and Applications , 2010 .

[38]  Maria Bolsinova,et al.  Response moderation models for conditional dependence between response time and response accuracy , 2017, The British journal of mathematical and statistical psychology.

[39]  Daniel Oberski,et al.  Hidden Markov Item Response Theory Models for Responses and Response Times , 2016, Multivariate behavioral research.

[40]  Jean-Paul Fox,et al.  Modeling of Responses and Response Times with the Package cirt , 2007 .

[41]  van der Linden,et al.  A hierarchical framework for modeling speed and accuracy on test items , 2007 .

[42]  Dandan Xu,et al.  A note on posterior predictive checks to assess model fit for incomplete data , 2016, Statistics in medicine.

[43]  Hua-Hua Chang,et al.  Sequential Detection of Compromised Items Using Response Times in Computerized Adaptive Testing , 2018, Psychometrika.

[44]  Jonathan P. Weeks,et al.  Using Response Time Data to Inform the Coding of Omitted Responses , 2016 .

[45]  M. Davier,et al.  Modeling Nonignorable Missing Data with Item Response Theory (IRT). Research Report. ETS RR-10-11. , 2010 .

[46]  Alison M. Mitchell,et al.  Improving the reliability of student scores from speeded assessments: an illustration of conditional item response theory using a computer-administered measure of vocabulary , 2015, Reading and writing.

[47]  Jean-Paul Fox,et al.  Evaluating cognitive theory: a joint modeling approach using responses and response times. , 2009, Psychological methods.

[48]  W. D. Linden,et al.  Conceptual Issues in Response-Time Modeling. , 2009 .

[49]  David B. Dunson,et al.  Bayesian Data Analysis , 2010 .

[50]  Eugenio Gonzalez,et al.  principles of multiple matrix booklet designs and parameter recovery in large-scale assessments , 2010 .

[51]  Pao-Kuei Wu,et al.  MISSING RESPONSES AND IRT ABILITY ESTIMATION: OMITS, CHOICE, TIME LIMITS, AND ADAPTIVE TESTING , 1996 .

[52]  J. Fox,et al.  Joint Modeling of Ability and Differential Speed Using Responses and Response Times , 2016, Multivariate behavioral research.

[53]  Colm O'Muircheartaigh,et al.  Symmetric pattern models: a latent variable approach to item non‐response in attitude scales , 1999 .

[54]  Jiqiang Guo,et al.  Stan: A Probabilistic Programming Language. , 2017, Journal of statistical software.

[55]  Steven L. Wise,et al.  An Application of Item Response Time: The Effort‐Moderated IRT Model , 2006 .

[56]  Barbara S. Plake,et al.  The Impact of Omitted Responses on the Accuracy of Ability Estimation in Item Response Theory , 2001 .

[57]  Holmes Finch,et al.  Estimation of Item Response Theory Parameters in the Presence of Missing Data , 2008 .

[58]  William N. Venables,et al.  Modern Applied Statistics with S , 2010 .

[59]  Andrew Gelman,et al.  Inference from Simulations and Monitoring Convergence , 2011 .

[60]  B. Muthén,et al.  How to Use a Monte Carlo Study to Decide on Sample Size and Determine Power , 2002 .

[61]  D. Rubin INFERENCE AND MISSING DATA , 1975 .

[62]  Steffi Pohl,et al.  Dealing With Omitted and Not-Reached Items in Competence Tests , 2014 .

[63]  Wim J. van der Linden,et al.  Bayesian Procedures for Identifying Aberrant Response-Time Patterns in Adaptive Testing , 2008 .

[64]  Steffi Pohl,et al.  Using Response Times to Model Not-Reached Items due to Time Limits , 2019, Psychometrika.

[65]  Rebecca Holman,et al.  Modelling non-ignorable missing-data mechanisms with item response theory models. , 2005, The British journal of mathematical and statistical psychology.