Measuring Socioeconomic Background in PISA: One Size Might not Fit all

As part of its flagship educational study – the Programme for International Student Assessment (PISA) – the Organisation for Economic Co-operation and Development (OECD) has undertaken extensive work to create an internationally relevant composite indicator aimed at measuring socioeconomic background. However, the degree to which a single measure of socioeconomic background is reliable and valid for all participating countries is not widely discussed. To fill this gap, the authors examine the home possessions index, which is a key component of PISA's socioeconomic indicator, and highlight a number of issues surrounding this index. In particular, they take a psychometric approach to investigating the reliability and some facets of the validity of the home possessions index in a number of participating PISA countries. Their findings suggest that there are notable concerns with the current index, including highly variable reliability by country, poor model-to-data consistency on a number of subscales, and evidence of poor cultural comparability. They couch their discussion in the context of educational and policy research and propose one possible method for improving these measures for participating countries.

[1]  E. Muraki,et al.  Chapter 3: Scaling Procedures in NAEP , 1992 .

[2]  M. Wolf,et al.  Measuring Socioeconomic Status , 2002, Assessment.

[3]  L. Cronbach.  Coefficient alpha and the internal structure of tests , 1951 .

[4]  H. Wagemaker.  Choices and trade‐offs: reply to McGaw , 2008 .

[5]  B. Mcgaw.  The role of the OECD in international comparative studies of achievement , 2008 .

[6]  M. R. Novick,et al.  Statistical Theories of Mental Test Scores. , 1971 .

[7]  G. Evans,et al.  Self-Regulation and the Income-Achievement Gap. , 2008 .

[8]  Bernard P. Veldkamp,et al.  A Model for the Construction of Country-Specific Yet Internationally Comparable Short-Form Marketing Scales , 2009, Mark. Sci..

[9]  Jagdip Singh,et al.  Measurement Issues in Cross-National Research , 1995 .

[10]  D. Sörbom.  A GENERAL METHOD FOR STUDYING DIFFERENCES IN FACTOR MEANS AND FACTOR STRUCTURE BETWEEN GROUPS , 1974 .

[11]  Robert J. Mislevy,et al.  Randomization-based inference about latent variables from complex samples , 1991 .

[12]  Martin R. West,et al.  *‘Every Catholic Child in a Catholic School’: Historical Resistance to State Schooling, Contemporary Private Competition and Student Achievement Across Countries , 2008, SSRN Electronic Journal.

[13]  P. Fayers.  Item Response Theory for Psychologists , 2004, Quality of Life Research.

[14]  P. Bentler,et al.  Cutoff criteria for fit indexes in covariance structure analysis : Conventional criteria versus new alternatives , 1999 .

[15]  J. Coleman.  THE CONCEPT OF EQUALITY OF EDUCATIONAL OPPORTUNITY. , 1968 .

[16]  David M. Shoemaker,et al.  Principles and procedures of multiple matrix sampling. , 1973 .

[17]  Emily Knowles,et al.  PISA 2009 : how does the social attainment gap in England compare with countries internationally? , 2012 .

[18]  Margaret Wu,et al.  Measurement, Sampling, and Equating Errors in Large‐Scale Assessments , 2010 .

[19]  W. Meredith.  Measurement invariance, factor analysis and factorial invariance , 1993 .

[20]  Leslie Rutkowski,et al.  Getting it ‘better’: the importance of improving background questionnaires in international large‐scale assessment , 2010 .

[21]  Robert J. Mislevy,et al.  Estimating Population Characteristics From Sparse Matrix Samples of Item Responses , 1992 .

[22]  Roderick P. McDonald,et al.  A note on the investigation of local and global identifiability , 1982 .

[23]  Leslie Rutkowski.  Sensitivity of Achievement Estimation to Conditioning Model Misclassification , 2014 .

[24]  Eugene G. Johnson,et al.  Scaling Procedures in NAEP , 1992 .

[25]  O. D. Duncan,et al.  Socioeconomic Background and Achievement. , 1974 .

[26]  Leslie Rutkowski.  The Impact of Missing Background Data on Subpopulation Estimation , 2011 .

[27]  André A. Rupp,et al.  An NCME Instructional Module on Booklet Designs in Large‐Scale Assessments of Student Achievement: Theory and Practice , 2009 .

[28]  N. Astone,et al.  Some Practical Guidelines for Measuring Youth's Race/Ethnicity and Socioeconomic Status , 1994 .

[29]  Henry May,et al.  A Multilevel Bayesian Item Response Theory Method for Scaling Socioeconomic Status in International Studies of Education , 2006 .

[30]  Gordon W. Cheung,et al.  Evaluating Goodness-of-Fit Indexes for Testing Measurement Invariance , 2002 .

[31]  Charles W. Mueller,et al.  Measures of socioeconomic status: Alternatives and recommendations. , 1981 .

[32]  Margaret Wu,et al.  A comparison of PISA and TIMSS 2003 achievement results in mathematics , 2009 .

[33]  Selcuk R. Sirin.  Socioeconomic Status and Academic Achievement: A Meta-Analytic Review of Research , 2005 .

[34]  K. White.  The relation between socioeconomic status and academic achievement. , 1982 .

[35]  Tom Schuller,et al.  Understanding the Social Outcomes of Learning , 2007 .

[36]  F. Chen.  Sensitivity of Goodness of Fit Indexes to Lack of Measurement Invariance , 2007 .

[37]  Barbara M. Byrne,et al.  Testing for Measurement and Structural Equivalence in Large-Scale Cross-Cultural Studies: Addressing the Issue of Nonequivalence , 2010 .

[38]  Wolfram Schulz,et al.  Measuring the socio-economic background of students and its effect on achievement in PISA 2000 and PISA 2003 , 2005 .

[39]  Jan de Leeuw,et al.  On the relationship between item response theory and factor analysis of discretized variables , 1987 .

[40]  Michael R. Mullen.  Diagnosing Measurement Equivalence in Cross-National Research , 1995 .

[41]  E. Grundy,et al.  The socioeconomic status of older adults: How should we measure it in studies of health inequalities? , 2001, Journal of epidemiology and community health.

[42]  Charalambos Vrasidas,et al.  Can Computer Use Hurt Science Achievement? The USA Results from PISA , 2003 .