Scaling, Linking, and Reporting in a Periodic Assessment System

A new entry in the testing lexicon is through-course summative assessment, a system consisting of components administered periodically during the academic year. As defined in the Race to the Top program, these assessments are intended to yield a yearly summative score for accountability purposes. They must provide for both individual and group proficiency estimates and allow for the measurement of growth. They must accommodate students who vary in their patterns of curricular exposure. Because they are meant to provide actionable information to teachers they must be instructionally sensitive, so item-operating characteristics can be expected to change relative to one another as a function of patterns of curricular exposure. This paper discusses methodology one can draw upon to tackle this ambitious collection of inferences. We consider a modeling framework that consists of an item response theory component and a population component, as in the National Assessment of Educational Progress, and show how performance and growth could be expressed in terms of expected performance on a market basket of tasks. We discuss conditions under which modeling simplifications might be possible and discuss studies that would be needed to fit models, estimate parameters, and evaluate data requirements.

[1]  R. Zwick.  Measurement Issues in State Achievement Comparisons , 2010 .

[2]  Leigh Burstein,et al.  Instructionally Sensitive Psychometrics: Application of a New IRT‐Based Detection Technique to Mathematics Achievement Test Items , 1991 .

[3]  Allan S. Cohen,et al.  Latent Transition Analysis With a Mixture Item Response Theory Measurement Model , 2010 .

[4]  Thomas M. Haladyna,et al.  THE ROLE OF INSTRUCTIONAL SENSITIVITY IN THE EMPIRICAL REVIEW OF CRITERION-REFERENCED TEST ITEMS , 1981 .

[5]  Kikumi K. Tatsuoka,et al.  Differential Item Functioning Resulting From The Use of Different Solution Strategies , 1988 .

[6]  Robert J. Mislevy,et al.  Estimation of Latent Group Effects , 1985 .

[7]  Isaac I. Bejar,et al.  Updating the Duplex Design for Test-Based Accountability in the Twenty-First Century , 2010 .

[8]  D. Rubin.  Formalizing Subjective Notions about the Effect of Nonrespondents in Sample Surveys , 1977 .

[9]  Frederic M. Lord,et al.  Statistical inferences about true scores , 1959 .

[10]  Margaret Wu.  The Role of Plausible Values in Large-Scale Surveys. , 2005 .

[11]  Susan E. Embretson,et al.  A multidimensional latent trait model for measuring learning and change , 1991 .

[12]  Robert J. Mislevy,et al.  Implications of Market-Basket Reporting for Achievement-Level Setting , 1998 .

[13]  Mark Wilson.  Saltus: A psychometric model of discontinuity in cognitive development. , 1989 .

[14]  Rebecca Zwick.  Effects of Item Order and Context on Estimation of NAEP Reading Proficiency , 1991 .

[15]  G. H. Fischer,et al.  Logistic latent trait models with linear constraints , 1983 .

[16]  Robert J. Mislevy,et al.  Randomization-based inference about latent variables from complex samples , 1991 .

[17]  Donald Hedeker,et al.  Full-information item bi-factor analysis , 1992 .

[18]  Matthias von Davier,et al.  Measuring Growth in a Longitudinal Large-Scale Assessment with a General Latent Variable Model , 2011 .

[19]  H.L.J. van der Maas,et al.  An IRT Model with a Parameter-Driven Process for Change , 2005 .

[20]  N. Thomas,et al.  Asymptotic Corrections for Multivariate Posterior Moments with Factored Likelihood Functions , 1993 .

[21]  Jon Cohen,et al.  Comparison of Partially Measured Latent Traits across Nominal Subgroups , 1999 .

[22]  Robert J. Mislevy,et al.  Estimating Population Characteristics From Sparse Matrix Samples of Item Responses , 1992 .

[23]  Morgan S. Polikoff.  Instructional Sensitivity as a Psychometric Property of Assessments , 2010 .

[24]  P. Rosenbaum,et al.  Conditional Association and Unidimensionality in Monotone Latent Variable Models , 1985 .

[25]  N. Thomas,et al.  Assessing Model Sensitivity of the Imputation Methods Used in the National Assessment of Educational Progress , 2000 .