Correcting for Test Score Measurement Error in ANCOVA Models for Estimating Treatment Effects

A common strategy for estimating treatment effects in observational studies using individual student-level data is analysis of covariance (ANCOVA) or hierarchical variants of it, in which outcomes (often standardized test scores) are regressed on pretreatment test scores, other student characteristics, and treatment group indicators. Measurement error in the prior test scores, which typically is both large and heteroscedastic, is regularly overlooked in empirical analyses and may erode the ability of regression models to adjust for student factors and may result in biased treatment effect estimates. We develop extensions of method-of-moments, Simulation-Extrapolation, and latent regression approaches to correcting for measurement error using the conditional standard errors of measure of test scores, and demonstrate their effectiveness relative to simpler alternatives using both simulation and a case study of teacher value-added effect estimation using longitudinal data from a large suburban school district.

[1]  Yi Shang.  Measurement Error Adjustment Using the SIMEX Method: An Application to Student Growth Percentiles , 2012 .

[2]  Jeffrey M. Woodbridge.  Econometric Analysis of Cross Section and Panel Data , 2002 .

[3]  Silvia Cagnone,et al.  A General Multivariate Latent Growth Model With Applications to Student Achievement , 2012 .

[4]  Jesse Rothstein,et al.  Student Sorting and Bias in Value-Added Estimation: Selection on Observables and Unobservables , 2009, Education Finance and Policy.

[5]  Cory Koedel,et al.  Test Measurement Error and Inference from Value-Added Models , 2012 .

[6]  M. J. Kolen,et al.  Conditional Standard Errors of Measurement for Scale Scores Using IRT , 1996 .

[7]  Raymond J. Carroll,et al.  Measurement error in nonlinear models: a modern perspective , 2006 .

[8]  F. Lord.  Applications of Item Response Theory To Practical Testing Problems , 1980 .

[9]  T. Wansbeek.  Measurement error and latent variables in econometrics , 2000 .

[10]  D. Hall.  Measurement Error in Nonlinear Models: A Modern Perspective , 2008 .

[11]  Martyn Plummer,et al.  JAGS: A program for analysis of Bayesian graphical models using Gibbs sampling , 2003 .

[12]  N. Laird.  Nonparametric Maximum Likelihood Estimation of a Mixing Distribution , 1978 .

[13]  Enrico Gori,et al.  Covariate Measurement Error Adjustment for Multilevel Models With Application to Educational Data , 2011 .

[14]  R. Hambleton,et al.  Handbook of Modern Item Response Theory , 1997 .

[15]  Andrew Thomas,et al.  WinBUGS - A Bayesian modelling framework: Concepts, structure, and extensibility , 2000, Stat. Comput..

[16]  Eugene G. Johnson,et al.  Scaling Procedures in NAEP , 1992 .

[17]  Thomas A Louis,et al.  Jump down to Document , 2022 .

[18]  Raj Chetty,et al.  Measuring the Impacts of Teachers I: Evaluating Bias in Teacher Value-Added Estimates , 2014 .

[19]  Joseph Kang,et al.  Demystifying Double Robustness: A Comparison of Alternative Strategies for Estimating a Population Mean from Incomplete Data , 2007, 0804.2958.

[20]  Mark D. Reckase.  The Real World is More Complicated than We Would Like , 2004 .

[21]  Joseph A. Martineau.  Distorting Value Added: The Use of Longitudinal, Vertically Scaled Student Achievement Data for Growth-Based, Value-Added Accountability , 2006 .

[22]  Daniel F. McCaffrey,et al.  Centering and Reference Groups for Estimates of Fixed Effects: Modifications to Felsdvreg , 2010 .

[23]  Dale Ballou,et al.  Test Scaling and Value-Added Measurement , 2009, Education Finance and Policy.

[24]  Kathleen G. Purdy,et al.  Likelihood analysis for errors-in-variables regression with replicate measurements , 1996 .

[25]  Stan Lipovetsky,et al.  Generalized Latent Variable Modeling: Multilevel,Longitudinal, and Structural Equation Models , 2005, Technometrics.

[26]  Sophia Rabe-Hesketh,et al.  Maximum Likelihood Estimation of Generalized Linear Models with Covariate Measurement Error , 2003 .

[27]  D. Rubin,et al.  The central role of the propensity score in observational studies for causal effects , 1983 .

[28]  Sophia Rabe-Hesketh,et al.  Correcting for covariate measurement error in logistic regression using nonparametric maximum likelihood estimation , 2003 .

[29]  Steven Andrew Culpepper,et al.  Using analysis of covariance (ANCOVA) with fallible covariates. , 2011, Psychological methods.

[30]  Tim R. Sass,et al.  Value-Added Models and the Measurement of Teacher Quality , 2006 .

[31]  Patrick Royston.  Flexible alternatives to the Cox model, and more , 2001 .

[32]  Daniel F. McCaffrey,et al.  Have We Identified Effective Teachers? Validating Measures of Effective Teaching Using Random Assignment. Research Paper. MET Project. , 2013 .

[33]  Henry Braun.  Using Student Progress to Evaluate Teachers: A Primer on Value-Added Models. Policy Information Perspective. , 2005 .

[34]  Roger E. Millsap,et al.  Invariance in Measurement and Prediction Revisited , 2007 .

[35]  M. Seltzer,et al.  Modeling Heterogeneity in Relationships Between Initial Status and Rates of Change: Treating Latent Variable Regression Coefficients as Random Coefficients in a Three-Level Hierarchical Model , 2010 .

[36]  K. Roeder,et al.  A Semiparametric Mixture Approach to Case-Control Studies with Errors in Covariables , 1996 .

[37]  KyungMann Kim,et al.  Contrasting treatment‐specific survival using double‐robust estimators , 2012 .

[38]  J LunnDavid,et al.  WinBUGS A Bayesian modelling framework , 2000 .

[39]  Anthony S. Bryk,et al.  Hierarchical Linear Models: Applications and Data Analysis Methods , 1992 .

[40]  Examination of a Simple Errors-in-Variables Model: A Demonstration of Marginal Maximum Likelihood , 2006 .

[41]  E. Muraki,et al.  Chapter 3: Scaling Procedures in NAEP , 1992 .

[42]  Gavin W. Fulmer.  Value‐added measures in education: What every educator needs to know , 2012 .

[43]  Andreas Ritter,et al.  Structural Equations With Latent Variables , 2016 .

[44]  Cory Koedel,et al.  Re-Examining the Role of Teacher Quality in the Educational Production Function. Working Paper 2007-03. , 2007 .

[45]  J. R. Cook,et al.  Simulation-Extrapolation Estimation in Parametric Measurement Error Models , 1994 .

[46]  H. White.  Maximum Likelihood Estimation of Misspecified Models , 1982 .

[47]  E. Lehmann.  Elements of large-sample theory , 1998 .

[48]  D. Rubin,et al.  Inference from Iterative Simulation Using Multiple Sequences , 1992 .

[49]  Jean-Paul Fox,et al.  Bayesian modeling of measurement error in predictor variables using item response theory , 2003 .

[50]  Roger E. Millsap,et al.  Invariance in measurement and prediction: Their relationship in the single-factor case. , 1997 .

[51]  J. R. Lockwood,et al.  Inverse probability weighting with error-prone covariates. , 2013, Biometrika.

[52]  D. Boyd,et al.  Measuring Test Measurement Error , 2012 .

[53]  Lynne Steuerle Schofield,et al.  The use of cognitive ability measures as explanatory variables in regression analysis , 2012, IZA journal of labor economics.

[54]  Daniel F. McCaffrey,et al.  Controlling for Individual Heterogeneity in Longitudinal Models, with Applications to Student Achievement , 2007, 0706.1401.

[55]  Derek C. Briggs,et al.  Due Diligence and the Evaluation of Teachers: A Review of the Value-Added Analysis Underlying the Effectiveness Rankings of Los Angeles Unified School District Teachers by the "Los Angeles Times". , 2011 .

[56]  Viswanath Devanarayan,et al.  Empirical simulation extrapolation for measurement error models with replicate measurements , 2002 .

[57]  Derek C. Briggs,et al.  The Sensitivity of Value-Added Modeling to the Creation of a Vertical Score Scale , 2009, Education Finance and Policy.

[58]  William D. Schafer.  Growth Scales as an Alternative to Vertical Scales , 2006 .