Multilevel multidimensional item response model with a multilevel latent covariate.

In a pre-test-post-test cluster randomized trial, one of the methods commonly used to detect an intervention effect involves controlling pre-test scores and other related covariates while estimating an intervention effect at post-test. In many applications in education, the total post-test and pre-test scores, ignoring measurement error, are used as response variable and covariate, respectively, to estimate the intervention effect. However, these test scores are frequently subject to measurement error, and statistical inferences based on the model ignoring measurement error can yield a biased estimate of the intervention effect. When multiple domains exist in test data, it is sometimes more informative to detect the intervention effect for each domain than for the entire test. This paper presents applications of the multilevel multidimensional item response model with measurement error adjustments in a response variable and a covariate to estimate the intervention effect for each domain.

[1]  H. Goldstein.  Multilevel Statistical Models , 2006 .

[2]  Jean-Paul Fox,et al.  Longitudinal measurement in health‐related surveys. A Bayesian joint growth model for multivariate ordinal responses , 2013, Statistics in medicine.

[3]  S. Raudenbush,et al.  Assessing Direct and Indirect Effects in Multilevel Designs with Latent Variables , 1999 .

[4]  Michael D. Toland,et al.  Effects of Blended Instructional Models on Math Performance , 2014 .

[5]  Harvey Goldstein,et al.  Modelling measurement errors and category misclassifications in multilevel models , 2008 .

[6]  Xiao-Li Meng,et al.  Posterior Predictive $p$-Values , 1994 .

[7]  Suzanne Jak,et al.  A Test for Cluster Bias: Detecting Violations of Measurement Invariance Across Clusters in Multilevel Data , 2013 .

[8]  S. Rabe-Hesketh,et al.  Generalized multilevel structural equation modeling , 2004 .

[9]  R. Vandenberg,et al.  A Review and Synthesis of the Measurement Invariance Literature: Suggestions, Practices, and Recommendations for Organizational Research , 2000 .

[10]  Steven Andrew Culpepper,et al.  Using analysis of covariance (ANCOVA) with fallible covariates. , 2011, Psychological methods.

[11]  Andrew C. Porter,et al.  Analysis of Covariance: Its Model and Use in Psychological Research. , 1987 .

[12]  Ruggero Bellio,et al.  Structural Modeling of Measurement Error in Generalized Linear Models with Rasch Measures as Covariates , 2011 .

[13]  Bradley P. Carlin,et al.  Bayesian measures of model complexity and fit , 2002 .

[14]  Roderick P. McDonald,et al.  A general model for two-level data with responses missing at random , 1993 .

[15]  S. Reise,et al.  Exploring the measurement invariance of psychological instruments: Applications in the substance use domain. , 1997 .

[16]  D. Rubin,et al.  Inference from Iterative Simulation Using Multiple Sequences , 1992 .

[17]  P. Boeck,et al.  Explanatory item response models : a generalized linear and nonlinear approach , 2004 .

[18]  A. Béguin,et al.  MCMC estimation of multidimensional IRT models , 1998 .

[19]  Sandip Sinharay,et al.  Assessing Fit of Unidimensional Item Response Theory Models Using a Bayesian Approach. , 2005 .

[20]  Daniel F. McCaffrey,et al.  Correcting for Test Score Measurement Error in ANCOVA Models for Estimating Treatment Effects , 2014 .

[21]  Cornelis A.W. Glas,et al.  A Bayesian Approach to Person Fit Analysis in Item Response Theory Models , 2003 .

[22]  S. Raudenbush.  Statistical analysis and optimal design for cluster randomized trials , 1997 .

[23]  Kristopher J Preacher,et al.  Reliability estimation in a multilevel confirmatory factor analysis framework. , 2014, Psychological methods.

[24]  Jean-Paul Fox,et al.  Modelling response error in school effectiveness research , 2004 .

[25]  Ulrich Trautwein,et al.  A 2 × 2 taxonomy of multilevel latent contextual models: accuracy-bias trade-offs in full and partial error correction models. , 2011, Psychological methods.

[26]  Terry E. Duncan,et al.  Multilevel Covariance Structure Analysis of Sibling Antisocial Behavior. , 1998 .

[27]  Tihomir Asparouhov,et al.  Item Response Modeling in Mplus: A Multi-Dimensional, Multi-Level, and Multi-Timepoint Example , 2013 .

[28]  Isaac I. Bejar,et al.  Biased assessment of program impact due to psychometric artifacts. , 1980 .

[29]  Jean-Paul Fox,et al.  Bayesian modeling of measurement error in predictor variables using item response theory , 2003 .

[30]  Kristopher J Preacher,et al.  Measurement Error Correction Formula for Cluster-Level Group Differences in Cluster Randomized and Observational Studies , 2016, Educational and psychological measurement.

[31]  Sandip Sinharay,et al.  How Often Do Subscores Have Added Value? Results from Operational and Simulated Data , 2010 .

[32]  M. Aitkin,et al.  Statistical Modelling Issues in School Effectiveness Studies , 1986 .

[33]  Kristopher J Preacher,et al.  Alternative Methods for Assessing Mediation in Multilevel Data: The Advantages of Multilevel SEM , 2011 .

[34]  J. de la Torre,et al.  A Comparison of Four Methods of IRT Subscoring , 2011 .

[35]  D. Rubin.  Bayesianly Justifiable and Relevant Frequency Calculations for the Applied Statistician , 1984 .

[36]  D. A. Kenny,et al.  The statistical analysis of data from small groups. , 2002, Journal of personality and social psychology.

[37]  A. Béguin,et al.  MCMC estimation and some model-fit analysis of multidimensional IRT models , 2001 .

[38]  Walter R. Gilks,et al.  Model checking and model improvement , 1995 .