Rankings and university performance: A conditional multidimensional approach

University rankings are the subject of a paradox: the more they are criticized by social scientists and experts on methodological grounds, the more they receive attention in policy making and the media. In this paper we attempt to give a contribution to the birth of a new generation of rankings, one that might improve on the current state of the art, by integrating new kind of information and using new ranking techniques. Our approach tries to overcome four main criticisms of university rankings, namely: monodimensionality; statistical robustness; dependence on university size and subject mix; lack of consideration of the input-output structure. We provide an illustration on European universities and conclude by pointing on the importance of investing in data integration and open data at European level both for research and for policy making.

[1]  Jamil Salmi,et al.  League Tables as Policy Instruments , 2015 .

[2]  Li Liu,et al.  Academic ranking of world universities using scientometrics  - A comment to the “Fatal Attraction” , 2005, Scientometrics.

[3]  Andrea Bonaccorsi,et al.  Knowledge, Diversity and Performance in European Higher Education: A Changing Landscape , 2014 .

[4]  K. Field,et al.  Measuring the efficiency of British universities: a multi‐period data envelopment analysis , 2004 .

[5]  Maureen McKelvey,et al.  How and why does cost efficiency of universities differ across European countries? : An explorative attempt using new microdata , 2014 .

[6]  R. Färe,et al.  Benefit and Distance Functions , 1996 .

[7]  Cinzia Daraio,et al.  The differentiation of the strategic profile of higher education institutions. New positioning indicators based on microdata , 2008, Scientometrics.

[8]  Léopold Simar,et al.  Advanced Robust and Nonparametric Methods in Efficiency Analysis: Methodology and Applications , 2007 .

[9]  Nian Cai Liu,et al.  The Academic Ranking of World Universities. , 2005 .

[10]  Ellen Hazelkorn,et al.  Rankings and the Battle for World-Class Excellence: Institutional Strategies and Policy Choices , 2009 .

[11]  H. Moed,et al.  The use of bibliometric data for the measurement of university research performance , 1985 .

[12]  Simon Marginson,et al.  To Rank or To Be Ranked: The Impact of Global Rankings in Higher Education , 2007 .

[13]  Rachel Bowden,et al.  Fantasy Higher Education: University and college league tables , 2000 .

[14]  Andrew C. Worthington,et al.  Efficiency, technology and productivity change in Australian universities, 1998–2003 , 2008 .

[15]  A. Saltelli,et al.  Rickety numbers: Volatility of university rankings and policy implications , 2011 .

[16]  Marc Meredith,et al.  Why Do Universities Compete in the Ratings Game? An Empirical Analysis of the Effects of the U.S. News and World Report College Rankings , 2004 .

[17]  Léopold Simar,et al.  How to measure the impact of environmental factors in a nonparametric production model , 2012, Eur. J. Oper. Res..

[18]  Jeffrey Evans Stake The Interplay between Law School Rankings, Reputations, and Resource Allocation: Ways Rankings Mislead , 2005 .

[19]  Simon Marginson,et al.  Global Position and Position Taking: The Case of Australia , 2007 .

[20]  Shawna Grosskopf,et al.  Centralized or decentralized control of school resources? A network model , 2015 .

[21]  Hurvey Leibenstein Allocative efficiency vs. X-Efficiency , 1966 .

[22]  P. W. Wilson,et al.  Statistical Inference in Nonparametric Frontier Models: The State of the Art , 1999 .

[23]  Sven Hemlin,et al.  Research on research evaluation , 1996 .

[24]  M. Farrell The Measurement of Productive Efficiency , 1957 .

[25]  Léopold Simar,et al.  Statistical Approaches for Non‐parametric Frontier Models: A Guided Tour , 2015 .

[26]  Donald F. Westerheijden,et al.  Disseminating the right information to the right audience: cultural determinants in the use (and misuse) of rankings , 2008 .

[27]  Stephen R. Porter,et al.  Institutional research productivity and the connection to average student quality and overall reputation , 2006 .

[28]  Harvey Goldstein,et al.  League Tables and Their Limitations: Statistical Issues in Comparisons of Institutional Performance , 1996 .

[29]  A. V. van Raan,et al.  Fatal attraction: Conceptual and methodological problems in the ranking of universities by bibliometric methods , 2005 .

[30]  M. Kenward,et al.  An Introduction to the Bootstrap , 2007 .

[31]  K. Webber,et al.  Measuring the Research Performance of Postsecondary Institutions , 2011 .

[32]  Lutz Bornmann,et al.  Do Universities or Research Institutions With a Specific Subject Profile Have an Advantage or a Disadvantage in Institutional Rankings? A Latent Class Analysis With Data From the SCImago Ranking , 2013, J. Assoc. Inf. Sci. Technol..

[33]  E. Hazelkorn The Impact of League Tables and Ranking Systems on Higher Education Decision Making , 2007 .

[34]  Jill Johnes,et al.  Efficiency and Productivity Change in the English Higher Education Sector from 1996/97 to 2004/5 , 2008 .

[35]  Shawna Grosskopf,et al.  Applied efficiency analysis in education , 2014 .

[36]  M. Abbott,et al.  The efficiency of Australian universities: a data envelopment analysis , 2003 .

[37]  Jill Johnes Measuring teaching efficiency in higher education: An application of data envelopment analysis to economics graduates from UK Universities 1993 , 2006, Eur. J. Oper. Res..

[38]  Léopold Simar,et al.  Directional distances and their robust versions: Computational and testing issues , 2013, Eur. J. Oper. Res..

[39]  Cláudia S. Sarrico,et al.  Subject mix and productivity in Portuguese universities , 2009, Eur. J. Oper. Res..

[40]  Cinzia Daraio,et al.  Universities and Strategic Knowledge Creation. Specialization and Performance in Europe , 2007 .

[41]  Léopold Simar,et al.  Measuring firm performance using nonparametric quantile-type distances , 2017 .

[42]  Dominique Deprins,et al.  Measuring Labor-Efficiency in Post Offices , 2006 .

[43]  Nicholas A. Bowman,et al.  U.S. News & World Report College Rankings: Modeling Institutional Effects on Organizational Reputation , 2009, American Journal of Education.

[44]  Cinzia Daraio,et al.  Universities and Strategic Knowledge Creation , 2007 .

[45]  Léopold Simar,et al.  Efficiency and economies of scale and scope in European universities. A directional distance approach , 2014 .

[46]  N. Dyke Twenty Years of University Report Cards , 2005 .

[47]  Razvan V. Florian,et al.  Irreproducibility of the results of the Shanghai academic ranking of world universities , 2007, Scientometrics.

[48]  Cláudia S. Sarrico,et al.  The European university landscape: A micro characterization based on evidence from the Aquameth project , 2011 .

[49]  Domingo Docampo Adjusted sum of institutional scores as an indicator of the presence of university systems in the ARWU ranking , 2011, Scientometrics.

[50]  Thed N. van Leeuwen,et al.  The Holy Grail of science policy: Exploring and combining bibliometric tools in search of scientific excellence , 2003, Scientometrics.

[51]  P. W. Wilson,et al.  Estimation and inference in two-stage, semi-parametric models of production processes , 2007 .

[52]  J. Salmi,et al.  League Tables as Policy Instruments: Uses and Misuses , 2007 .

[53]  P. W. Wilson,et al.  Inferences from Cross-Sectional, Stochastic Frontier Models , 2009 .

[54]  J. Salmi The challenge of establishing world-class universities , 2009 .

[55]  Nian Cai Liu,et al.  The Story of Academic Ranking of World Universities , 2015 .

[56]  Cláudia S. Sarrico,et al.  Using DEA for planning in UK universities—an institutional perspective , 2000, J. Oper. Res. Soc..

[57]  Jill Johnes,et al.  Efficiency and Mergers in English Higher Education 1996/97 to 2008/9: Parametric and Non‐Parametric Estimation of the Multi‐Input Multi‐Output Distance Function , 2014 .

[58]  Lutz Bornmann,et al.  Multilevel-statistical reformulation of citation-based university rankings: The Leiden ranking 2011/2012 , 2013, J. Assoc. Inf. Sci. Technol..

[59]  Gualberto Buela-Casal,et al.  Comparative study of international academic rankings of universities , 2007, Scientometrics.

[60]  Vicente Safón What do global university rankings really measure? The search for the X factor and the X entity , 2013, Scientometrics.

[61]  Léopold Simar,et al.  Efficiency and economies of scale and specialization in European universities: A directional distance approach , 2015, J. Informetrics.

[62]  Ellen Hazelkorn,et al.  Rankings and the battle for world-class excellence , 2009 .

[63]  R. Färe,et al.  Efficiency and Productivity: Malmquist and More , 2008 .