Getting it ‘better’: the importance of improving background questionnaires in international large‐scale assessment

In addition to collecting achievement data, international large‐scale assessment programmes gather auxiliary information from students and schools regarding the context of teaching and learning. In an effort to clarify some of the opacity surrounding international large‐scale assessment programmes and the potential problems associated with less than optimal background questionnaires, this paper outlines how auxiliary student background data influence the black box of achievement score construction. This discussion is supplemented with a number of empirical examples that point to possible threats to accurate achievement estimation, including missing data treatment, poor scale reliability, and questionnaire respondents’ misunderstanding or inaccurate answers to the questions.

[1]  Robert J. Mislevy,et al.  Randomization-based inference about latent variables from complex samples , 1991 .

[2]  Torsten Husén,et al.  Policy Impact of IEA Research , 1987, Comparative Education Review.

[3]  A. Beaton,et al.  Secondary Analysis of the Timss Data , 2002 .

[4]  Katherine A. Magnuson.  Maternal education and children's academic achievement during middle childhood. , 2007, Developmental psychology.

[5]  David M. Shoemaker,et al.  Principles and procedures of multiple matrix sampling. , 1973 .

[6]  Peter J. Denning,et al.  A nation at risk: the imperative for educational reform , 1983, CACM.

[7]  John Cresswell,et al.  Assessing scientific, reading and mathematical literacy : a framework for PISA 2006 , 2006 .

[8]  Robert J. Mislevy,et al.  Estimating Population Characteristics From Sparse Matrix Samples of Item Responses , 1992 .

[9]  Eugene G. Johnson.  The Design of the National Assessment of Educational Progress , 1992 .

[10]  P. G. Moore,et al.  Data Uncertainties and Least Squares Regression , 1972 .

[11]  J. Hox,et al.  The validity of comparative educational studies , 2005 .

[12]  Eugene G. Johnson,et al.  Scaling Procedures in NAEP , 1992 .

[13]  Christine Y. O'Sullivan,et al.  TIMSS 2007 Assessment Frameworks. , 2005 .

[14]  Rebecca Zwick,et al.  Overview of the National Assessment of Educational Progress , 1992 .

[15]  Frederic M. Lord,et al.  ESTIMATING NORMS BY ITEM SAMPLING , 1961 .

[16]  Jens Dolin.  PISA - An Example of the Use and Misuse of Large-Scale Comparative Tests , 2007 .

[17]  Sotiria Grek,et al.  Governing by numbers: the PISA ‘effect’ in Europe , 2009 .

[18]  P. Davis-Kean,et al.  The influence of parent education and family income on child achievement: the indirect role of parental expectations and the home environment. , 2005, Journal of family psychology : JFP : journal of the Division of Family Psychology of the American Psychological Association.