A Semiparametric Model for Jointly Analyzing Response Times and Accuracy in Computerized Testing

The item response times (RTs) collected from computerized testing represent an underutilized type of information about items and examinees. In addition to knowing the examinees’ responses to each item, we can investigate the amount of time examinees spend on each item. Current models for RTs mainly focus on parametric models, which have the advantage of conciseness, but may suffer from reduced flexibility to fit real data. We propose a semiparametric approach, specifically, the Cox proportional hazards model with a latent speed covariate to model the RTs, embedded within the hierarchical framework proposed by van der Linden to model the RTs and response accuracy simultaneously. This semiparametric approach combines the flexibility of nonparametric modeling and the brevity and interpretability of the parametric modeling. A Markov chain Monte Carlo method for parameter estimation is given and may be used with sparse data obtained by computerized adaptive testing. Both simulation studies and real data analysis are carried out to demonstrate the applicability of the new model.

[1]  Bradley A. Hanson,et al.  Development and Calibration of an Item Response Model That Incorporates Response Time , 2005 .

[2]  P. McCullagh Regression Models for Ordinal Data , 1980 .

[3]  L. Tierney Markov Chains for Exploring Posterior Distributions , 1994 .

[4]  Edward E. Roskam,et al.  Models for Speed and Time-Limit Tests , 1997 .

[5]  R. Duncan Luce,et al.  Response Times: Their Role in Inferring Elementary Mental Organization , 1986 .

[6]  Deborah L. Schnipke,et al.  Representing Response-Time Information in Item Banks. Law School Admission Council Computerized Testing Report. LSAC Research Report Series. , 1999 .

[7]  Brent Bridgeman,et al.  Effects of Differentially Time-Consuming Tests on Computer-Adaptive Test Scores , 2004 .

[8]  Richard J. Patz,et al.  A Straightforward Approach to Markov Chain Monte Carlo Methods for Item Response Models , 1999 .

[9]  Carl de Boor,et al.  A Practical Guide to Splines , 1978, Applied Mathematical Sciences.

[10]  C. R. Deboor,et al.  A practical guide to splines , 1978 .

[11]  Jochen Ranger,et al.  A flexible latent trait model for response times in tests , 2012 .

[12]  J. Wolfowitz,et al.  On a Test Whether Two Samples are from the Same Population , 1940 .

[13]  R. Gray A Bayesian analysis of institutional effects in a multicenter cancer clinical trial. , 1994, Biometrics.

[14]  J. Fox,et al.  A Box-Cox normal model for response times. , 2009, The British journal of mathematical and statistical psychology.

[15]  M. Kosorok,et al.  A latent variable model for discrete multivariate psychometric waiting times , 1999 .

[16]  Hua-Hua Chang,et al.  Psychometrics Behind Computerized Adaptive Testing , 2015, Psychometrika.

[17]  David Thissen,et al.  Timed Testing: An Approach Using Item Response Theory , 1983 .

[18]  John K Kruschke,et al.  Bayesian data analysis. , 2010, Wiley interdisciplinary reviews. Cognitive science.

[19]  Denny Borsboom,et al.  Cognitive psychology meets psychometric theory: on the relation between process models for decision making and latent variable models for individual differences. , 2011, Psychological review.

[20]  C Brownie,et al.  A rank-based mixed model approach to multisite clinical trials. , 1992, Biometrics.

[21]  Barnes Discussion of the Paper , 1961, Public health papers and reports.

[22]  Ulrich Mansmann,et al.  A semiparametric Bayesian proportional hazards model for interval censored data with frailty effects , 2009, BMC medical research methodology.

[23]  D. Clayton,et al.  Multivariate generalizations of the proportional hazards model , 1985 .

[24]  Identifiers California,et al.  Annual Meeting of the National Council on Measurement in Education , 1998 .

[25]  P Gustafson,et al.  Large hierarchical Bayesian analysis of multivariate survival data. , 1997, Biometrics.

[26]  Xuming He,et al.  Monotone B-Spline Smoothing , 1998 .

[27]  van der Linden,et al.  A hierarchical framework for modeling speed and accuracy on test items , 2007 .

[28]  Krista Breithaupt,et al.  Detecting Differential Speededness in Multistage Testing , 2007 .

[29]  J. Kalbfleisch Non‐Parametric Bayesian Analysis of Survival Time Data , 1978 .

[30]  Rob J Hyndman,et al.  Mixed Model-Based Hazard Estimation , 2002 .

[31]  M. W. Tate Individual Differences in Speed of Response in Mental Test Materials of Varying Degrees of Difficulty , 1948, Educational and psychological measurement.

[32]  Hua-Hua Chang,et al.  The linear transformation model with frailties for the analysis of item response times. , 2013, The British journal of mathematical and statistical psychology.

[33]  김수진,et al.  Latent variable model에 의한 바이러스 유형 분석 , 2005 .

[34]  Wim J. van der Linden,et al.  Bayesian Procedures for Identifying Aberrant Response-Time Patterns in Adaptive Testing , 2008 .

[35]  B. Gibson,et al.  Using hazard functions to assess changes in processing capacity in an attentional cuing paradigm. , 2004, Journal of experimental psychology. Human perception and performance.

[36]  T. Ortner,et al.  A latent trait model for response times on tests employing the proportional hazards model. , 2012, The British journal of mathematical and statistical psychology.

[37]  D J Sargent,et al.  A general framework for random effects survival analysis in the Cox proportional hazards setting. , 1998, Biometrics.

[38]  D.,et al.  Regression Models and Life-Tables , 2022 .

[39]  Stephen J. Boies,et al.  Components of attention. , 1971 .

[40]  N. D. Verhelst,et al.  A Logistic Model for Time-Limit Tests , 1992 .

[41]  D G Clayton,et al.  A Monte Carlo method for Bayesian inference in frailty models. , 1991, Biometrics.

[42]  John B. Willett,et al.  It’s About Time: Using Discrete-Time Survival Analysis to Study Duration and the Timing of Events , 1993 .

[43]  Kjell A. Doksum,et al.  An extension of partial likelihood methods for proportional hazard models to general transformation models , 1987 .

[44]  Niels Keiding,et al.  Statistical Models Based on Counting Processes , 1993 .

[45]  M. Friedman Piecewise Exponential Models for Survival Data with Covariates , 1982 .

[46]  J. Kalbfleisch,et al.  Marginal likelihoods based on Cox's regression and life model , 1973 .

[47]  Deborah L. Schnipke,et al.  Modeling Item Response Times With a Two-State Mixture Model: A New Method of Measuring , 1997 .

[48]  Jeffrey N. Rouder,et al.  A hierarchical bayesian statistical framework for response time distributions , 2003 .

[49]  J WIM,et al.  A HIERARCHICAL FRAMEWORK FOR MODELING SPEED AND ACCURACY ON TEST ITEMS , 2007 .

[50]  Cornelis A.W. Glas,et al.  Statistical Tests of Conditional Independence Between Responses and/or Response Times on Test Items , 2010 .

[51]  D. Ruppert,et al.  Bayesian adaptive B-spline estimation in proportional hazards frailty models , 2010 .

[52]  Hua-Hua Chang,et al.  Utilizing Response Time Distributions for Item Selection in CAT , 2012 .