Measuring Efficiency: A Comparison of Multilevel Modelling and Data Envelopment Analysis in the Context of Higher Education

Data envelopment analysis (DEA) and multilevel modelling (MLM) are applied to a data set of 54,564 graduates from UK universities in 1993 to assess whether the choice of technique affects the measurement of universities’ performance. A methodology developed by Thanassoulis and Portela (2002; Education Economics, 10(2), pp. 183–207) allows each individual's DEA efficiency score to be decomposed into two components: one attributable to the university at which the student studied and the other attributable to the individual student. From the former component, a measure of each institution's teaching efficiency is derived and compared to the university effects from various multilevel models. The comparisons are made within four broad subjects: pure science, applied science, social science and arts. The results show that the rankings of universities derived from the DEA efficiencies which measure the universities’ own performance (i.e., having excluded the efforts of the individuals) are not strongly correlated with the university rankings derived from the university effects of the multilevel models. The data were also used to perform a university-level DEA. The university efficiency scores derived from these DEAs are largely unrelated to the scores from the individual-level DEAs, confirming a result from a smaller data set (Johnes, 2006a; European Journal of Operational Research, forthcoming). However, the university-level DEAs provide efficiency scores which are generally strongly related to the university effects of the multilevel models.

[1]  J. Johnes.  Inter-university variations in undergraduate non-completion rates: A statistical analysis by subject of study , 1997 .

[2]  Robin Naylor,et al.  Dropping out of university: A statistical analysis of the probability of withdrawal for UK university students , 2001 .

[3]  Emmanuel Thanssoulis,et al.  Setting Achievement Targets for School Children , 1999 .

[4]  Geraint Johnes,et al.  The costs of multi-product organizations and the heuristic evaluation of industrial structure , 1998 .

[5]  P. Dolton,et al.  Degree Class and Pass Rates: An Inter-University Comparison. , 1985 .

[6]  Kenneth V. Greene,et al.  The Effects of Monitoring and Competition on Public Education Outputs: A Stochastic Frontier Approach , 2002 .

[7]  J. Johnes,et al.  The 1989 Research Selectivity Exercise: A Statistical Analysis of Differences in Research Rating between Universities at the Cost Centre Level , 1992 .

[8]  Gary Madden,et al.  Measuring Public Sector Efficiency: A Study of Economics Departments at Australian Universities , 1997 .

[9]  Rafael A. Cuesta,et al.  Mergers and technical efficiency in Spanish savings banks: A stochastic distance function approach , 2002 .

[10]  Michael Barrow,et al.  Measuring local education authority performance: A frontier approach , 1991 .

[11]  Steven C. Deller,et al.  Production efficiency in elementary education: The case of Maine public schools , 1993 .

[12]  Ali Emrouznejad,et al.  An exploratory analysis of the cost structure of Higher Education in England , 2005 .

[13]  Ernest Rudd,et al.  A comparison between the results achieved by women and men studying for first degrees in British universities , 1984 .

[14]  J. Johnes,et al.  Undergraduate non-completion rates: Differences between UK universities , 1989 .

[15]  Cláudia S. Sarrico,et al.  Using DEA for planning in UK universities—an institutional perspective , 2000, J. Oper. Res. Soc..

[16]  M. Abbott,et al.  The efficiency of Australian universities: a data envelopment analysis , 2003 .

[17]  John E. Beasley,et al.  Comparing university departments , 1990 .

[18]  Jeremy P. Smith,et al.  Higher education outcomes, graduate employment and university performance indicators , 2004 .

[19]  J. Johnes.  Measuring Teaching Efficiency in Higher Education : an Application of Data Envelopment Analysis to Graduates from UK Universities 1993 , 2003 .

[20]  V. Barnett,et al.  A Study of the Relation between G. C. E. and Degree Results , 1963 .

[21]  Kenneth J. Rowe,et al.  Multilevel Modelling in School Effectiveness Research. , 1996 .

[22]  Roel Bosker,et al.  Multilevel analysis : an introduction to basic and advanced multilevel modeling , 1999 .

[23]  Min Yang,et al.  Progress from GCSE to A and AS Level: Institutional and gender differences, and trends over time , 2001 .

[24]  Harvey Goldstein,et al.  Educational Performance Indicators and LEA League Tables , 1988 .

[25]  Jill Johnes,et al.  Performance Indicators in Higher Education: Uk Universities , 1991 .

[26]  J. Johnes,et al.  The first destination of new graduates: comparisons between universities , 1989 .

[27]  Risto Lahdelma,et al.  Data envelopment analysis: visualizing the results , 1995 .

[28]  J. Johnes,et al.  Degree quality: An investigation into differences between UK universities , 1987 .

[29]  Timothy Rodgers,et al.  Measuring value added in higher education: a proposal , 1995 .

[30]  R. Kapur.  Student wastage at Edinburgh university. I. Factors related to failure and dropout , 1972 .

[31]  G. Johnes.  THE MANAGEMENT OF UNIVERSITIES SCOTTISH ECONOMIC SOCIETY = ROYAL BANK OF SCOTLAND ANNUAL LECTURE , 1999 , 1999 .

[32]  H. Goldstein.  Multilevel Statistical Models , 2006 .

[33]  N. Avkiran.  INVESTIGATING TECHNICAL AND SCALE EFFICIENCIES OF AUSTRALIAN UNIVERSITIES THROUGH DATA ENVELOPMENT ANALYSIS , 2001 .

[34]  C. Lovell,et al.  A survey of frontier production functions and of their relationship to efficiency measurement , 1980 .

[35]  P. Stevens,et al.  A Stochastic Frontier Analysis of English and Welsh Universities , 2005 .

[36]  J. Rushton.  Book Review: Degrees of Excellence: the academic achievement game , 1978 .

[37]  Robin Naylor,et al.  Determinants of Degree Performance in UK Universities: A Statistical Analysis of the 1993 Student Cohort , 2001 .

[38]  Timothy Coelli,et al.  An Introduction to Efficiency and Productivity Analysis , 1997 .

[39]  Massimiliano Bratti,et al.  Does the Choice of University Matter? A Study of the Differences across UK Universites in Life Sciences Students' Degree Performance , 2002 .

[40]  The university performance of mature students , 1975 .

[41]  Emmanuel Thanassoulis,et al.  Decomposing school and school-type efficiency , 2001, Eur. J. Oper. Res..

[42]  Jill Johnes,et al.  Performance assessment in higher education in Britain , 1996 .

[43]  A. Parikh,et al.  Headmasters' Reports, Admissions and Academic Performance in Social Sciences. , 1983 .

[44]  A. U.S.,et al.  FORMULATION AND ESTIMATION OF STOCHASTIC FRONTIER PRODUCTION FUNCTION MODELS , 2001 .

[45]  Antreas D. Athanassopoulos,et al.  Assessing the Comparative Efficiency of Higher Education Institutions in the UK by the Means of Data Envelopment Analysis , 1997 .

[46]  School Outcomes: Sharing the Responsibility Between Pupil and School1 , 2002 .

[47]  Kevin Sear,et al.  The correlation between a level grades and degree results in England and Wales , 1983 .

[48]  J. Beasley.  Determining Teaching and Research Efficiencies , 1995 .

[49]  Cláudia S. Sarrico,et al.  Data envelopment analysis and university selection , 1997 .

[50]  G. Johnes.  The Management of Universities: President's Lecture Delivered at Annual General Meeting of the Scottish Economic Society 6-8th April 1999 , 1999 .

[51]  J. Johnes.  The Potential Effects of Wider Access to Higher Education on Degree Quality , 1992 .

[52]  W. Montague,et al.  Academic selection criteria and subsequent performance , 1990, Medical education.

[53]  Deb Ghosh,et al.  Measuring the determinants of quality in UK higher education: a multinomial logit approach , 2001 .

[54]  J. Johnes,et al.  Non-completion of a Degree Course and Its Effect on the Subsequent Experience of Non-completers in the Labour Market. , 1991 .

[55]  Harvey Goldstein,et al.  League Tables and Their Limitations: Statistical Issues in Comparisons of Institutional Performance , 1996 .

[56]  Jill Johnes,et al.  Apples and oranges: The aggregation problem in publications analysis , 2005, Scientometrics.

[57]  Noel Entwistle,et al.  Degrees of excellence: The academic achievement game , 1977 .

[58]  R. Green,et al.  AN EXPERIMENT IN THE USE OF DATA ENVELOPMENT ANALYSIS FOR EVALUATING THE EFFICIENCY OF UK UNIVERSITY DEPARTMENTS OF ACCOUNTING , 1988 .

[59]  Jeremy P. Smith,et al.  Graduate Employability: Policy and Performance in Higher Education in the UK , 2000 .

[60]  H. Goldstein,et al.  Using Examination Results as Indicators of School and College Performance , 1996 .

[61]  Jill Johnes,et al.  Measuring the Research Performance of UK Economics Departments: An Application of Data Envelopment Analysis , 1993 .

[62]  Peter Chalos,et al.  An Examination of Budgetary Inefficiency in Education Using Data Envelopment Analysis , 1997 .

[63]  Jill Johnes.  DEA and its Application to the Measurement of Efficiency in Higher Education , 2006 .

[64]  Variability of degree results in geography in United Kingdom universities 1973–90: Preliminary results and policy implications , 1994 .

[65]  Jyrki Wallenius,et al.  Value efficiency analysis of academic research , 1998, Eur. J. Oper. Res..

[66]  Jill Johnes.  Measuring teaching efficiency in higher education: An application of data envelopment analysis to economics graduates from UK Universities 1993 , 2006, Eur. J. Oper. Res..

[67]  Robert Crouchley,et al.  Stochastic frontier estimation of a CES cost function: the case of higher education in Britain , 2002 .