Are there test administrator effects in large-scale educational assessments? Using cross-classified multilevel analysis to probe for effects on mathematics achievement and sample attrition

Abstract. In large-scale educational assessments such as the Third International Mathematics and Sciences Study (TIMSS) or the Program for International Student Assessment (PISA), sizeable numbers of test administrators (TAs) are needed to conduct the assessment sessions in the participating schools. TA training sessions are run and administration manuals are compiled with the aim of ensuring standardized, comparable, assessment situations in all student groups. To date, however, there has been no empirical investigation of the effectiveness of these standardizing efforts. In the present article, we probe for systematic TA effects on mathematics achievement and sample attrition in a student achievement study. Multilevel analyses for cross-classified data using Markov Chain Monte Carlo (MCMC) procedures were performed to separate the variance that can be attributed to differences between schools from the variance associated with TAs. After controlling for school effects, only a very small, nonsignificant p...

[1]  William J. Browne,et al.  Bayesian and likelihood-based methods in multilevel modeling 1 A comparison of Bayesian and likelihood-based methods for fitting multilevel models , 2006 .

[2]  David B. Dunson,et al.  Bayesian Data Analysis , 2010 .

[3]  Colm O'Muircheartaigh,et al.  A multilevel exploration of the role of interviewers in survey non‐response , 1999 .

[4]  Risto Lethonen.  Multilevel Statistical Models (3rd ed.) , 2005 .

[5]  J. Gill.  Hierarchical Linear Models , 2005 .

[6]  Wilfried Bos,et al.  TIMSS/III Dritte Internationale Mathematik- und Naturwissenschaftsstudie — Mathematische und naturwissenschaftliche Bildung am Ende der Schullaufbahn , 2000 .

[7]  R. Rosenthal,et al.  Covert communication in the psychological experiment. , 1967, Psychological bulletin.

[8]  H. Goldstein.  Multilevel Statistical Models , 2006 .

[9]  R. Fildes.  Journal of the Royal Statistical Society (B): Gary K. Grunwald, Adrian E. Raftery and Peter Guttorp, 1993, “Time series of continuous proportions”, 55, 103–116.☆ , 1993 .

[10]  Geert Loosveldt,et al.  The Effects of Interviewer and Respondent Characteristics on Response Behavior in Panel Surveys , 2001 .

[11]  S. R. Searle,et al.  Generalized, Linear, and Mixed Models , 2005 .

[12]  Bradley P. Carlin,et al.  Markov Chain Monte Carlo conver-gence diagnostics: a comparative review , 1996 .

[13]  R. Groves,et al.  Survey Errors and Survey Costs. , 1991 .

[14]  O. Köller,et al.  Wege zur Hochschulreife in Baden-Württemberg: TOSCA - eine Untersuchung an allgemein bildenden und beruflichen Gymnasien , 2004 .

[15]  M. Clement,et al.  PREVENTION AND TREATMENT , 1944 .

[16]  Bradley P. Carlin,et al.  Bayesian measures of model complexity and fit , 2002 .

[17]  R. Rosenthal,et al.  Experimenter effects in behavioral research , 1968 .

[18]  Geoff N Masters,et al.  Measuring student knowledge and skills : the PISA 2000 assessment of reading, mathematical and scientific literacy , 2000 .

[19]  H. Goldstein,et al.  Efficient Analysis of Mixed Hierarchical and Cross-Classified Random Structures Using a Multilevel Model , 1994 .

[20]  J. Newsom.  Another Side to Caregiving , 1999 .

[21]  A. Beaton.  Mathematics Achievement in the Primary School Years. IEA's Third International Mathematics and Science Study (TIMSS). , 1996 .

[22]  G. Casella,et al.  Explaining the Gibbs Sampler , 1992 .

[23]  Michael O. Martin,et al.  Mathematics and Science Achievement in the Final Year of Secondary School: IEA's Third International Mathematics and Science Study (TIMSS). , 1998 .

[24]  R. Rosenthal.  Unintended effects of the clinician in clinical interaction , 1969 .

[25]  S. Chib,et al.  Understanding the Metropolis-Hastings Algorithm , 1995 .

[26]  Rainer Schnell,et al.  Separating interviewer and sampling-point effects , 2003 .

[27]  Raymond J. Adams,et al.  PISA 2000 technical report , 2002 .

[28]  Robert M. Groves,et al.  Survey Errors and Survey Costs: Groves/Survey Errors , 2005 .

[29]  H. Goldstein,et al.  Multilevel Models in Educational and Social Research. , 1989 .

[30]  S. R. Searle,et al.  Generalized, Linear, and Mixed Models: McCulloch/Generalized, Linear, and Mixed Models , 2005 .

[31]  Roel Bosker,et al.  Multilevel analysis : an introduction to basic and advanced multilevel modeling , 1999 .

[32]  S. T. Buckland,et al.  An Introduction to the Bootstrap. , 1994 .

[33]  D. Tritchler,et al.  Communicating probabilistic information to cancer patients: is there 'noise' on the line? , 1991, Social science & medicine.

[34]  Timothy J. Robinson,et al.  Multilevel Analysis: Techniques and Applications , 2002 .