How to measure research efficiency in higher education? Research grants vs. publication output

ABSTRACT An ongoing debate in the literature on the efficiency of higher education institutions concerns the indicator for research output for use in empirical analysis. While several studies have chosen to use the number of publications as this indicator, others rely on the amount of research grants. The present study investigates whether both measures lead to a similar assessment of universities. In addition, the number of publications belonging to the 10% and 1% most frequently cited papers in the corresponding subject category and publication year are evaluated. We show that there is a high correlation of efficiency values between the estimations using these indicators. This correlation is slightly higher when the efficiency values result from a data envelopment analysis than when they are determined with a stochastic frontier analysis. The results of this study provide helpful guidelines for researchers evaluating the efficiency of universities and are valueable for decision-makers in science policy.

[1]  Joanna Wolszczak-Derlacz,et al.  Efficiency of European public higher education institutions: a two-stage multicountry approach , 2011, Scientometrics.

[2]  Lutz Bornmann,et al.  Efficiency of research performance and the glass researcher , 2016, J. Informetrics.

[3]  Kristof De Witte,et al.  Efficiency in education: a review of literature and a way forward , 2017, J. Oper. Res. Soc..

[4]  Dag W. Aksnes,et al.  A macro study of self-citation , 2003, Scientometrics.

[5]  Loet Leydesdorff,et al.  The use of percentiles and percentile rank classes in the analysis of bibliometric data: Opportunities and limits , 2012, J. Informetrics.

[6]  J. Johnes.  Data Envelopment Analysis and Its Application to the Measurement of Efficiency in Higher Education. , 2006 .

[7]  S. Rijcke,et al.  Bibliometrics: The Leiden Manifesto for research metrics , 2015, Nature.

[8]  Ulrich Schmoch,et al.  How to use indicators to measure scientific performance: a balanced approach , 2010 .

[9]  Marvin A. Titus,et al.  Examining Production Efficiency in Higher Education: The Utility of Stochastic Frontier Analysis , 2016 .

[10]  Erik E. Lehmann,et al.  Approaching effects of the economic crisis on university efficiency: a comparative study of Germany and Italy , 2018 .

[11]  Dorothea Jansen,et al.  Drittmittel als Performanzindikator der Wissenschaftlichen Forschung , 2007 .

[12]  Subal C. Kumbhakar,et al.  Technical efficiency in competing panel data models: a study of Norwegian grain farming , 2014 .

[13]  J. Fredericks Volkwein,et al.  The Cost Structure of American Research Universities , 1991 .

[14]  W. Baumol,et al.  Contestable Markets and the Theory of Industry Structure , 1982 .

[15]  Lutz Bornmann,et al.  What is societal impact of research and how can it be assessed? a literature survey , 2013, J. Assoc. Inf. Sci. Technol..

[16]  Gerhard Kempkes,et al.  Do Institutions Matter for University Cost Efficiency? Evidence from Germany , 2008 .

[17]  Carolyn J. Heinrich.  Outcomes–Based Performance Management in the Public Sector: Implications for Government Accountability and Effectiveness , 2002 .

[18]  Tommaso Agasisti,et al.  The transient and persistent efficiency of Italian and German universities: a stochastic frontier analysis , 2019, Applied Economics.

[19]  Mehdi Rhaiem,et al.  Measurement and determinants of academic research efficiency: a systematic review of the evidence , 2017, Scientometrics.

[20]  Brigitte Tiefenthaler,et al.  Counting quality? The Czech performance-based research funding system , 2015 .

[21]  P. Stevens,et al.  A Stochastic Frontier Analysis of English and Welsh Universities , 2005 .

[22]  Manjulika Koshal,et al.  Economies of scale and scope in higher education: a case of comprehensive universities , 1999 .

[23]  Ali Emrouznejad,et al.  An exploratory analysis of the cost structure of Higher Education in England , 2005 .

[24]  Adela García Aracil,et al.  The differential effects of competitive funding on the production frontier and the efficiency of universities , 2016 .

[25]  M. Sunder,et al.  The German excellence initiative and efficiency change among universities, 2001-2011 , 2016 .

[26]  Lutz Bornmann,et al.  Measuring the societal impact of research , 2012, EMBO reports.

[27]  Jill Johnes,et al.  Measuring the Research Performance of UK Economics Departments: An Application of Data Envelopment Analysis , 1993 .

[28]  N. Cox,et al.  A Note on the Concordance Correlation Coefficient , 2002 .

[29]  G. Kempkes,et al.  The efficiency of German universities–some evidence from nonparametric and parametric methods , 2010 .

[30]  K. Tone,et al.  The efficiency of higher education institutions in England revisited: comparing alternative measures , 2017 .

[31]  Dominic Orr,et al.  Performance‐based funding as an instrument of competition in German higher education , 2007 .

[32]  D. Aksnes,et al.  Researchers’ perceptions of citations , 2009 .

[33]  Giovanni Abramo,et al.  A farewell to the MNCS and like size-independent indicators , 2016, J. Informetrics.

[34]  J. W. Lewis,et al.  A note on concordance correlation coefficient. , 2000, PDA journal of pharmaceutical science and technology.

[35]  Andrew C. Worthington,et al.  Economies of scale and scope in Australian higher education , 2011 .

[36]  Eleonora Freddi.  Do People Avoid Morally Relevant Information? Evidence from the Refugee Crisis , 2017, Review of Economics and Statistics.

[37]  L. Butler,et al.  Explaining Australia’s increased share of ISI publications—the effects of a funding formula based on publication counts , 2003 .

[38]  Vietnamese Manufacturing Firms.  STOCHASTIC FRONTIER ANALYSIS OF , 2014 .

[39]  Joanicjusz Nazarko,et al.  Application of DEA method in efficiency evaluation of public higher education institutions , 2014 .

[40]  Heike Wetzel,et al.  Competing in the Higher Education Market: Empirical Evidence for Economies of Scale and Scope in German Higher Education Institutions , 2014 .

[41]  S. Gralka.  Persistent inefficiency in the higher education sector: evidence from Germany , 2018 .

[42]  Torben Schubert,et al.  Empirical observations on New Public Management to increase efficiency in public research--Boon or bane? , 2009 .

[43]  Mika Nieminen,et al.  University research funding and publication performance--An international comparison , 2010 .

[44]  K. Wohlrabe,et al.  Wie effizient forschen Universitäten in Deutschland, deren Zukunftskonzepte im Rahmen der Exzellenzinitiative ausgezeichnet wurden? , 2018 .

[45]  G. Battese,et al.  Frontier production functions, technical efficiency and panel data: With application to paddy farmers in India , 1992 .

[46]  Sabine Gralka,et al.  Stochastic frontier analysis in higher education: A systematic review , 2018 .

[47]  Peter Woelert,et al.  Follow the money? How Australian universities replicate national performance-based funding mechanisms , 2018, Research Evaluation.

[48]  Andrew C. Worthington,et al.  Efficiency, technology and productivity change in Australian universities, 1998–2003 , 2008 .

[49]  Julia Sonnenburg,et al.  Zur Effizienz der Hochschulen in den ost- und westdeutschen Flächenländern , 2015 .

[50]  Timothy Coelli,et al.  An Introduction to Efficiency and Productivity Analysis , 1997 .

[51]  G. Johnes.  COSTS AND INDUSTRIAL STRUCTURE IN CONTEMPORARY BRITISH HIGHER EDUCATION , 1997 .

[52]  Stefan Hornbostel,et al.  Third party funding of German universities. An indicator of research activity? , 2001, Scientometrics.

[53]  M. Abbott,et al.  The efficiency of Australian universities: a data envelopment analysis , 2003 .

[54]  Susanne Warning,et al.  The economic analysis of universities : strategic groups and positioning , 2007 .

[55]  A. Charnes,et al.  Some Models for Estimating Technical and Scale Inefficiencies in Data Envelopment Analysis , 1984 .

[56]  Roberto Verganti,et al.  Evaluating the performance of academic departments: an analysis of research-related output efficiency , 2012 .

[57]  Robert Crouchley,et al.  Stochastic frontier estimation of a CES cost function: the case of higher education in Britain , 2002 .

[58]  Selim Zaim,et al.  Measuring the relative efficiency of quality management practices in Turkish public and private universities , 2013, J. Oper. Res. Soc..

[59]  M. Woodhall,et al.  Economics of Education: A Review , 1987 .

[60]  G. Johnes,et al.  Heterogeneity and the evaluation of efficiency: the case of Italian universities , 2010 .

[61]  Differences in cost structure and the evaluation of efficiency: the case of German universities , 2011 .

[62]  International Comparisons Of Performance In The Provision Of Public Services , 2004 .

[63]  Carla Haelermans,et al.  Comparing Efficiency of Public Universities Among European Countries: Different Incentives Lead to Different Performances , 2014 .

[64]  D. Aigner,et al.  P. Schmidt, 1977,?Formulation and estimation of stochastic frontier production function models,? , 1977 .

[65]  Emmanuel Thanassoulis,et al.  Costs and efficiency of higher education institutions in England: a DEA analysis , 2011, J. Oper. Res. Soc..

[66]  A. Worthington.  An Empirical Survey of Frontier Efficiency Measurement Techniques in Education , 2001 .

[67]  Barry Bozeman,et al.  Technology transfer and public policy: a review of research and theory , 2000 .

[68]  Günter Fandel,et al.  On the performance of universities in North Rhine-Westphalia, Germany: Government's redistribution of funds judged using DEA efficiency measures , 2007, Eur. J. Oper. Res..

[69]  Geraint Johnes,et al.  Beyond frontiers: comparing the efficiency of higher education decision‐making units across more than one country , 2009 .