Assessing the quality of institutions’ rankings obtained through multilevel linear regression models

The aim of this paper is to assess the quality of the ranking of institutions obtained with multilevel techniques in presence of different model misspecifications and data structures. Through a Monte Carlo simulation study, we find that it is quite hard to obtain a reliable ranking of the whole effectiveness distribution while, under various experimental conditions, it is possible to identify institutions with extreme performances. Ranking quality increases with increasing intra class correlation coefficient and/or overall sample size. Furthermore, multilevel models where the between and within cluster components of first-level covariates are distinguished perform significantly better than both multilevel models where the two effects are set to be equal and the fixed effect models.

[1]  Kerstin Schneider,et al.  Fair Ranking of Teachers Running Title: "Fair Ranking of Teachers" , 2006 .

[2]  H. Pan,et al.  A Multilevel Analysis of School Examination Results , 1993 .

[3]  Sophia Rabe-Hesketh,et al.  Generalized latent variable models: multilevel, longitudinal, and structural equation models , 2004 .

[4]  Cora J. M. Maas,et al.  Robustness issues in multilevel regression analysis , 2004 .

[5]  S. Rabe-Hesketh,et al.  Latent Variable Modelling: A Survey * , 2007 .

[6]  Alessandra Petrucci,et al.  Analysis of university course evaluations: from descriptive measures to multilevel models , 2004 .

[7]  Sophia Rabe-Hesketh,et al.  Multilevel and longitudinal modeling using Stata. 2nd edition , 2008 .

[8]  Melanie Hughes,et al.  Are “Failing” Schools Really Failing? Using Seasonal Comparison to Evaluate School Effectiveness , 2008 .

[9]  Helen F. Ladd,et al.  Implementing value-added measures of school effectiveness: getting the incentives right , 2002 .

[10]  H. Goldstein,et al.  The Graphical Presentation of a Collection of Means , 1995 .

[11]  L. Wößmann,et al.  Efficiency and equity of European education and training policies , 2006 .

[12]  G. Maddala,et al.  THE USE OF VARIANCE COMPONENTS MODELS IN POOLING CROSS SECTION AND TIME SERIES DATA , 1971 .

[13]  H. Goldstein,et al.  The limitations of using school league tables to inform school choice , 2009 .

[14]  N. Horton.  Multilevel and Longitudinal Modeling Using Stata , 2006 .

[15]  James Algina,et al.  An Empirical Comparison of Statistical Models for Value-Added Assessment of School Performance , 2004 .

[16]  Harvey Goldstein,et al.  League Tables and Their Limitations: Statistical Issues in Comparisons of Institutional Performance , 1996 .

[17]  Tom Marshall,et al.  Performance league tables: the NHS deserves better , 2002, BMJ : British Medical Journal.

[18]  M. Aitkin,et al.  Statistical Modelling Issues in School Effectiveness Studies , 1986 .

[19]  C. Rampichini.  Measurement error in multilevel models with sample cluster means , 2009 .

[20]  A. Aassve,et al.  Dynamic multi-level analysis of households' living standards and poverty: Evidence from Vietnam , 2007 .

[21]  Stephen W. Raudenbush,et al.  The Estimation of School Effects , 1995 .

[22]  J. Kalbfleisch,et al.  Between- and within-cluster covariate effects in the analysis of clustered data. , 1998, Biometrics.