Applying the Pseudo-Panel Approach to International Large-Scale Assessments: A Methodology for Analyzing Subpopulation Trend Data

Applying the Pseudo-Panel Approach to International Large-Scale Assessments: A Methodology for Analyzing Subpopulation Trend Data Dissertation by Martin Hooper Advisor: Ina V. S. Mullis, Ph.D. TIMSS and PIRLS assess representative samples of students at regular intervals, measuring trends in student achievement and student contexts for learning. Because individual students are not tracked over time, analysis of international large-scale assessment data is usually conducted cross-sectionally. Gustafsson (2007) proposed examining the data longitudinally by analyzing relationships between country-level trends in background constructs and trends in student achievement. Through longitudinal analysis of international large-scale assessment data, it becomes possible to mitigate some of the confounding factors in the analysis. This dissertation extends this country-level approach to subpopulations within countries. Adapting a pseudo-panel approach from the econometrics literature (Deaton, 1985), the proposed approach creates subpopulations by grouping students based on demographic characteristics, such as gender or parental education. Following grouping, the subpopulations with the same demographic characteristics are linked across cycles and the aggregated subpopulation means are treated as panel data and analyzed through longitudinal data analysis techniques. As demonstrated herein the primary advantages of the subpopulation approach are that it allows for analysis of subgroup differences, and it captures within-country relationships in the data that are not possible to analyze at country level. Illustrative analysis examines the relationship between early literacy activities and PIRLS reading achievement using PIRLS 2001 and PIRLS 2011 data. Results from the subpopulation approach are compared with student-level and country-level cross-sectional results as well as country-level longitudinal results. In addition, within-country analysis examines the subpopulation-level relationship between early literacy activities and PIRLS reading achievement, multiple group analysis compares regression coefficient estimates between boys and girls and across parental education subgroups, and mediation analysis examines the extent that partaking in early literacy activities can explain differences between boys and girls in PIRLS reading achievement.

[1]  Christopher Winship,et al.  Counterfactuals and Causal Inference: Methods and Principles for Social Research , 2007 .

[2]  Mauro Mediavilla,et al.  The Evolution of Educational Inequalities in Spain: Dynamic Evidence from Repeated Cross-Sections , 2016 .

[3]  Mauro Mediavilla,et al.  Double Toil and Trouble: Grade Retention and Academic Performance , 2016 .

[4]  T. Cook,et al.  Quasi-experimentation: Design & analysis issues for field settings , 1979 .

[5]  W. S. Robinson.  Ecological correlations and the behavior of individuals. , 1950, International journal of epidemiology.

[6]  C. Nelson,et al.  Lessons from neuroscience research for understanding causal links between family and neighborhood characteristics and educational outcomes , 2011 .

[7]  Reading behaviour from adolescence to early adulthood , 2012 .

[8]  D. Freedman,et al.  A solution to the ecological inference problem , 1997 .

[9]  G. Masters.  A rasch model for partial credit scoring , 1982 .

[10]  K. Bollen,et al.  A General Panel Model with Random and Fixed Effects: A Structural Equations Approach , 2010, Social forces; a scientific medium of social study and interpretation.

[11]  Leigh Burstein,et al.  Estimation from Grouped Observations , 1974 .

[12]  Michael O. Martin,et al.  PIRLS 2016 Assessment Framework. 2nd Edition. , 2015 .

[13]  D. Rubin.  Estimating causal effects of treatments in randomized and nonrandomized studies. , 1974 .

[14]  A. Deaton.  The Analysis of Household Surveys : A Microeconometric Approach to Development Policy , 1997 .

[15]  D. Rubin,et al.  Assessing Sensitivity to an Unobserved Binary Covariate in an Observational Study with Binary Outcome , 1983 .

[16]  J. P. Morgan,et al.  Design and Analysis: A Researcher's Handbook , 2005, Technometrics.

[17]  G. Kraaykamp,et al.  The effects of parental reading socialization and early school involvement on children's academic performance: a panel study of primary school pupils in the Netherlands , 2011 .

[18]  T. Husén.  INTERNATIONAL STUDY OF ACHIEVEMENT IN MATHEMATICS, A COMPARISON OF TWELVE COUNTRIES, VOLUME I. , 1967 .

[19]  M. Montgomery,et al.  Child health, breast-feeding, and survival in Malaysia: a random-effects logit approach. , 1986, Journal of the American Statistical Association.

[20]  Marno Verbeek,et al.  A Guide to Modern Econometrics , 2000 .

[21]  P. Allison.  Fixed Effects Regression Models , 2009 .

[22]  Jan-Eric Gustafsson,et al.  Is computer availability at home causally related to reading achievement in grade 4? A longitudinal difference in differences approach to IEA data from 1991 to 2006 , 2016 .

[23]  G. Kraaykamp,et al.  Parental media socialization and educational attainment: Resource or disadvantage? , 2010 .

[24]  J. LeFevre,et al.  Parental involvement in the development of children's reading skill: a five-year longitudinal study. , 2002, Child development.

[25]  Martin Browning,et al.  A Profitable Approach to Labor Supply and Commodity Demands over the Life-Cycle , 1985 .

[26]  J. Gustafsson.  Causal inference in educational effectiveness research: a comparison of three methods to investigate effects of homework on student achievement 1 , 2013 .

[27]  Anthony S. Bryk,et al.  Hierarchical Linear Models: Applications and Data Analysis Methods , 1992 .

[28]  K. Magnuson,et al.  Investing in Preschool Programs , 2013, The journal of economic perspectives : a journal of the American Economic Association.

[29]  J. Kmenta.  Mostly Harmless Econometrics: An Empiricist's Companion , 2010 .

[30]  W. Shadish,et al.  Experimental and Quasi-Experimental Designs for Generalized Causal Inference , 2001 .

[31]  K. Bollen,et al.  Fixed and Random Effects in Panel Data Using Structural Equations Models , 2008 .

[32]  Costas Meghir,et al.  LABOR SUPPLY AND INTERTEMPORAL SUBSTITUTION , 1993 .

[33]  J. Pearl.  Causality: Models, Reasoning and Inference , 2000 .

[34]  Psychometric Framework for Modeling Parental Involvement and Reading Literacy , 2016 .

[35]  E. Hanushek,et al.  Does School Autonomy Make Sense Everywhere? Panel Estimates from PISA , 2011, SSRN Electronic Journal.

[36]  Jeffrey M. Woodbridge.  Econometric Analysis of Cross Section and Panel Data , 2002 .

[37]  Ludger Woessmann,et al.  Econometric methods for causal evaluation of education policies and practices: a non‐technical guide , 2009, SSRN Electronic Journal.

[38]  J. Gustafsson,et al.  The Impact of School Climate and Teacher Quality on Mathematics Achievement: A Difference-in-Differences Approach , 2016 .

[39]  E. Hanushek,et al.  School Resources and Student Achievement: A Review of Cross-Country Economic Research , 2017 .

[40]  Ina V. S. Mullis,et al.  TIMSS 2011 International Results in Science. , 2012 .

[41]  B. Baltagi,et al.  Econometric Analysis of Panel Data , 2020, Springer Texts in Business and Economics.

[42]  S. Suter.  Meaningful differences in the everyday experience of young American children , 2005, European Journal of Pediatrics.

[43]  Paul J. Devereux,et al.  Small sample bias in synthetic cohort models of labor supply , 2007 .

[44]  R. Moffitt.  Identification and estimation of dynamic models with a time series of repeated cross-sections , 1993 .

[45]  Causal inferences with large scale assessment data: using a validity framework , 2016 .

[46]  Angus Deaton.  Panel data from time series of cross-sections , 1985 .

[47]  Elizabeth Dhuey,et al.  The Analysis of International Large-Scale Assessments to Address Causal Questions in Education Policy , 2017 .

[48]  Sven E. Wilson,et al.  A Lot More to Do: The Sensitivity of Time-Series Cross-Section Analyses to Simple Alternative Specifications , 2007, Political Analysis.

[49]  D. Mackinnon.  Introduction to Statistical Mediation Analysis , 2008 .

[50]  A. Ginsburg.  A Study of the Usefulness of RCTs in the What Works Clearinghouse , 2016 .

[51]  M. Rowe.  A longitudinal investigation of the role of quantity and quality of child-directed speech in vocabulary development. , 2012, Child development.

[52]  P. Allison.  Fixed Effects Regression Methods for Longitudinal Data Using SAS® , 2005 .

[53]  L. Baker,et al.  BEGINNING READERS' MOTIVATION FOR READING IN RELATION TO PARENTAL BELIEFS AND HOME READING EXPERIENCES , 2002 .

[54]  Marno Verbeek,et al.  Can cohort data be treated as genuine panel data? , 1992 .

[55]  C. Glymour,et al.  STATISTICS AND CAUSAL INFERENCE , 1985 .

[56]  P. Allison,et al.  CHANGE SCORES,FIXED EFFECTS,AND RANDOM COMPONENTS : A STRUCTURAL EQUATION APPROACH , 1998 .

[57]  P. Foy,et al.  Scaling the TIMSS and PIRLS 2011 Achievement Data , 2011 .

[58]  N. Lazar,et al.  The ASA Statement on p-Values: Context, Process, and Purpose , 2016 .

[59]  Anne Fernald,et al.  Talking to Children Matters , 2013, Psychological science.

[60]  Marno Verbeek,et al.  Pseudo Panels and Repeated Cross-Sections , 2007 .

[61]  Pierre Foy,et al.  PIRLS 2011 User Guide for the International Database. Supplement 4: PIRLS 2011 Sampling Stratification Information. , 2013 .

[62]  Gianfranco De Simone,et al.  Render unto primary the things which are primary's: Inherited and fresh learning divides in Italian lower secondary education , 2013 .

[63]  Roger A. Sugden,et al.  Multiple Imputation for Nonresponse in Surveys , 1988 .

[64]  P. Robinson Joseph.  Causal inference and comparative analysis with large-scale assessment data , 2013 .

[65]  Pam Sammons,et al.  Effects of the Home Learning Environment and Preschool Center Experience upon Literacy and Numeracy Development in Early Primary School , 2008 .

[66]  Introduction to special issue on quasi-causal methods , 2016 .