Linking via Pseudo‐Equivalent Group Design: Methodological Considerations and an Application to the PISA and PIAAC Assessments

This article presents the pseudo‐equivalent group approach and discusses how it can enhance the quality of linking in the presence of nonequivalent groups. The pseudo‐equivalent group approach allows to achieve pseudo‐equivalence using propensity score reweighting techniques. We use it to perform linking to establish scale concordance between two assessments. The article presents Monte‐Carlo simulations and a real data application based on data from the Survey of Adult Skills (PIAAC) and the Programme for International Student Assessment (PISA). Monte‐Carlo simulations suggest that the pseudo‐equivalent group design is particularly useful whenever there is a large overlap across the two groups with respect to balancing variables and when the correlation between such variables and ability is medium or high. The example based on PISA and PIAAC data indicates that the approach can provide reasonable accurate linking that can be used for group‐level comparisons.

[1]  J. R. Lockwood,et al.  Inverse probability weighting with error-prone covariates. , 2013, Biometrika.

[2]  S. Haberman.  Pseudo-Equivalent Groups and Linking , 2015 .

[3]  Shelby J. Haberman,et al.  Adjustment by Minimum Discriminant Information , 1984 .

[4]  David Kaplan,et al.  Data fusion with international large scale assessments: a case study using the OECD PISA and TALIS surveys , 2013 .

[5]  Margaret Wu.  The Role of Plausible Values in Large-Scale Surveys. , 2005 .

[6]  Eugene G. Johnson,et al.  Linking the National Assessment of Educational Progress (NAEP) and the Third International Mathematics and Science Study (TIMSS): Eighth-Grade Results. Research and Development Report. , 1998 .

[7]  D. Rubin,et al.  The central role of the propensity score in observational studies for causal effects , 1983 .

[8]  Linking TIMSS and NAEP assessments to evaluate international trends in achievement , 2017 .

[9]  Comparison of the Performance of Ontario Students on the OSSLT/TPCL and the PISA 2009 Reading Assessment , 2012 .

[10]  F. Borgonovi,et al.  An international comparison of students' ability to endure fatigue and maintain motivation during a low-stakes test , 2016 .

[11]  E. Hanushek,et al.  The Role of International Assessments of Cognitive Skills in the Analysis of Growth and Development , 2013 .

[12]  R. Hambleton,et al.  How Do Other Countries Measure Up to the Mathematics Achievement Levels on the National Assessment of Educational Progress? , 2009 .

[13]  Samuel A. Livingston,et al.  INVESTIGATING DIFFERENCES IN EXAMINEE PERFORMANCE BETWEEN COMPUTER-BASED AND HANDWRITTEN ESSAYS , 2004 .

[14]  M. Frölich.  Finite-Sample Properties of Propensity-Score Matching and Weighting Estimators , 2004 .

[15]  Marie Wiberg,et al.  Kernel Equating Under the Non-Equivalent Groups With Covariates Design , 2015, Applied psychological measurement.

[16]  Jonathan Kelley,et al.  Scholarly culture: How books in adolescence enhance adult literacy, numeracy and technology skills in 31 societies. , 2019, Social science research.

[17]  R. Brennan,et al.  Test Equating, Scaling, and Linking , 2004 .

[18]  R. Linn,et al.  Utility and Validity of NAEP Linking Efforts. , 2009 .

[19]  P. Austin.  An Introduction to Propensity Score Methods for Reducing the Effects of Confounding in Observational Studies , 2011, Multivariate behavioral research.

[20]  Margaret Wu,et al.  A comparison of PISA and TIMSS 2003 achievement results in mathematics , 2009 .

[21]  D. Rubin.  Estimating causal effects of treatments in randomized and nonrandomized studies. , 1974 .

[22]  Effect on Equating Results of Matching Samples on an Anchor Test. , 1990 .

[23]  P. Austin.  The International Journal of Biostatistics Type I Error Rates , Coverage of Confidence Intervals , and Variance Estimation in Propensity-Score Matched Analyses , 2011 .