32 The Statistical Procedures Used in National Assessment of Educational Progress: Recent Developments and Future Directions

The National Assessment of Educational Progress (NAEP) is an ongoing survey of the performance of the school students in the U.S. in a number of subject areas, including reading, writing, and mathematics. This chapter discusses in detail the statistical model, the current estimation technique, possible alternatives, and future directions of psychometric research in NAEP. The reporting method used in NAEP relies on a complex latent regression model and sampling weights with some of the assessments involving more than a hundred thousand examinees. Estimating the model parameters, therefore, is not straightforward for NAEP data, and neither is reporting of the results. For example, in order to generate maximum (marginal) likelihood estimates of the parameters of the latent regression model, multivariate (of up to five dimensions) integrals must be evaluated. Also, computing standard errors is not straightforward because of the complicated nature of the model and the complex sampling scheme. Still, the current estimation method used in NAEP performs respectably. However, the method has often been criticized recently. A number of researchers have suggested alternatives to the current statistical techniques in NAEP.

[1]  E. Muraki.  A Generalized Partial Credit Model: Application of an EM Algorithm , 1992 .

[2]  Rebecca Zwick,et al.  Overview of the National Assessment of Educational Progress , 1992 .

[3]  D. Rubin,et al.  Multiple Imputation for Nonresponse in Surveys , 1989 .

[4]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[5]  Lyle V. Jones.  The Nation's Report Card: Evolution and Perspectives , 2004 .

[6]  Albert E. Beaton,et al.  Expanding the New Design: The NAEP 1985-86 Technical Report. , 1988 .

[7]  N. Thomas,et al.  The role of secondary covariates when estimating latent trait population distributions , 2002 .

[8]  R. Mislevy.  Estimating latent distributions , 1984 .

[9]  David B. Dunson,et al.  Bayesian Data Analysis , 2010 .

[10]  J. Rao.  Small Area Estimation , 2003 .

[11]  Eugene G. Johnson,et al.  Considerations and Techniques for the Analysis of NAEP Data , 1988 .

[12]  A. Zellner.  An Efficient Method of Estimating Seemingly Unrelated Regressions and Tests for Aggregation Bias , 1962 .

[13]  Matthias von Davier,et al.  APPLICATION OF THE STOCHASTIC EM METHOD TO LATENT REGRESSION MODELS , 2004 .

[14]  K. Rust,et al.  Population Inferences and Variance Estimation for NAEP Data , 1992 .

[15]  M. R. Novick,et al.  Statistical Theories of Mental Test Scores. , 1971 .

[16]  Stephen W. Raudenbush,et al.  Synthesizing Results from the Trial State Assessment , 1999 .

[17]  Raymond J. Adams,et al.  Multilevel Item Response Models: An Approach to Errors in Variables Regression , 1997 .

[18]  B. Efron.  The jackknife, the bootstrap, and other resampling plans , 1987 .

[19]  John E. Dennis,et al.  Numerical methods for unconstrained optimization and nonlinear equations , 1983, Prentice Hall series in computational mathematics.

[20]  Karen Draney,et al.  Objective measurement : theory into practice , 1992 .

[21]  Matthew S. Johnson,et al.  A BAYESIAN HIERARCHICAL MODEL FOR LARGE-SCALE EDUCATIONAL SURVEYS: AN APPLICATION TO THE NATIONAL ASSESSMENT OF EDUCATIONAL PROGRESS , 2004 .

[22]  Kenneth G. Manton,et al.  “Equivalent Sample Size” and “Equivalent Degrees of Freedom” Refinements for Inference Using Survey Weights under Superpopulation Models , 1992 .

[23]  Jon Cohen,et al.  Comparison of Partially Measured Latent Traits across Nominal Subgroups , 1999 .

[24]  N. Thomas,et al.  Asymptotic Corrections for Multivariate Posterior Moments with Factored Likelihood Functions , 1993 .

[25]  Robert J. Mislevy,et al.  Estimation of Latent Group Effects , 1985 .