The Methodology of PISA: Past, Present, and Future

International large-scale assessments (ILSAs) such as PISA, TIMSS and PIRLS represent the state-of-the art methodologies of sampling theory, survey research design, and psychometrics. The purpose of this chapter is to provide an accessible overview of the methodologies that underlie ILSAs generally, with a specific focus on PISA. We focus on the PISA sampling design, issues of translation and adaptability, the methodologies that are used for the development of cognitive tests, and finally the methodologies used to assess the material included in the context questionnaire. This chapter closes with a discussion of future methodological developments.

[1]  Roderick J. A. Little,et al.  Statistical Analysis with Missing Data: Little/Statistical Analysis with Missing Data , 2002 .

[2]  David M. Shoemaker,et al.  Principles and procedures of multiple matrix sampling. , 1973 .

[3]  Ramona B. Swaine.  Theory of Probability (Vols. 1 and 2) , 1981 .

[4]  S. James Press,et al.  Subjective and objective Bayesian statistics : principles, models, and applications , 2003 .

[5]  Jeremy E. Oakley,et al.  Uncertain Judgements: Eliciting Experts' Probabilities , 2006 .

[6]  Michael O. Martin,et al.  TIMSS 2015 Assessment Frameworks. , 2013 .

[7]  A. Purves.  The Evolution of the IEA: A Memoir , 1987, Comparative Education Review.

[8]  N. Jude.  The Assessment of Learning Contexts in PISA , 2016 .

[9]  M. R. Novick,et al.  Statistical Theories of Mental Test Scores. , 1971 .

[10]  D. Kaplan.  Causal inference with large‑scale assessments in education from a Bayesian perspective: a review and synthesis , 2016, Large-scale Assessments in Education.

[11]  Kaplan David,et al.  Analyzing International Large-Scale Assessment Data within a Bayesian Framework , 2013 .

[12]  Pierre Simon Laplace.  Essai philosophique sur les probabilités , 1921 .

[13]  Jaap Scheerens,et al.  The Foundations of Educational Effectiveness , 1997 .

[14]  J. Berger.  The case for objective Bayesian analysis , 2006 .

[15]  E. Klieme,et al.  Assessing Contexts of Learning , 2016 .

[16]  Bengt Muthén,et al.  Bayesian structural equation modeling: a more flexible representation of substantive theory. , 2012, Psychological methods.

[17]  Surajit Ray,et al.  BIC and Alternative Bayesian Information Criteria in the Selection of Structural Equation Models , 2014, Structural equation modeling : a multidisciplinary journal.

[18]  Lars Tummers,et al.  Facing off with Scylla and Charybdis: a comparison of scalar, partial, and the novel possibility of approximate measurement invariance , 2013, Front. Psychol..

[19]  E. Klieme,et al.  Dimensions of Context Assessment , 2016 .

[20]  D. Kaplan,et al.  Bayesian Statistics for the Social Sciences , 2014 .

[21]  E. S. Pearson,et al.  ON THE USE AND INTERPRETATION OF CERTAIN TEST CRITERIA FOR PURPOSES OF STATISTICAL INFERENCE PART I , 1928 .

[22]  D. Rubin,et al.  Statistical Analysis with Missing Data. , 1989 .

[23]  T. A. Warm.  Weighted likelihood estimation of ability in item response theory , 1989 .

[24]  Scientific.  International Standard Classification of Education, ISCED 1997 , 2003 .

[25]  John Cresswell,et al.  Assessing scientific, reading and mathematical literacy : a framework for PISA 2006 , 2006 .

[26]  T. Bayes.  An essay towards solving a problem in the doctrine of chances , 2003 .

[27]  Robert J. Mislevy,et al.  Estimating Population Characteristics From Sparse Matrix Samples of Item Responses , 1992 .

[28]  Bengt Muthén,et al.  Multiple-Group Factor Analysis Alignment , 2014 .

[29]  André A. Rupp,et al.  An NCME Instructional Module on Booklet Designs in Large‐Scale Assessments of Student Achievement: Theory and Practice , 2009 .

[30]  James M. Dickey,et al.  Scientific Reporting and Personal Probabilities: Student's Hypothesis , 1973 .

[31]  L. J. Savage,et al.  The Foundations of Statistics , 1955 .

[32]  Sabine Meinck,et al.  The relationship between students’ use of ICT for social communication and their computer and information literacy , 2016 .

[33]  D. Rubin.  Estimating causal effects of treatments in randomized and nonrandomized studies. , 1974 .