Funnel plots for visualizing uncertainty in the research performance of institutions

Research performance values are not certain. Performance indexes should therefore be accompanied by uncertainty measures, to establish whether the performance of a unit is truly outstanding and not the result of random fluctuations. In this work we focus on the evaluation of research institutions on the basis of average individual performance, where uncertainty is inversely related to the number of research staff. We utilize the funnel plot, a tool originally developed in meta-analysis, to measure and visualize the uncertainty in the performance values of research institutions. As an illustrative example, we apply the funnel plot to represent the uncertainty in the assessed research performance for Italian universities active in biochemistry.

[1]  Cinzia Daraio,et al.  Exploring size and agglomeration effects on public research productivity , 2005, Scientometrics.

[2]  R. Mutz,et al.  Entwicklung eines Hochschul-Rankingverfahrens mittels Mixed-Rasch-Modell und Mehrebenenanalyse am Beispiel der Psychologie , 2007 .

[3]  Giovanni Abramo,et al.  Assessment of sectoral aggregation distortion in research productivity measurements , 2008 .

[4]  Alejandro Márquez Jiménez.  Academic Ranking of World Universities 2008 , 1969 .

[5]  Y. Rosseel.  Latent Variable Analysis , 2015 .

[6]  Lutz Bornmann,et al.  The substantive and practical significance of citation impact differences between institutions: Guidelines for the analysis of percentiles using effect sizes and confidence intervals , 2014, ArXiv.

[7]  Giovanni Abramo,et al.  The importance of accounting for the number of co-authors and their order when assessing research performance at the individual level in the life sciences , 2018, J. Informetrics.

[8]  S. Bird,et al.  Performance indicators: good, bad, and ugly , 2004 .

[9]  Claudio Castellano,et al.  Universality of citation distributions: Toward an objective measure of scientific impact , 2008, Proceedings of the National Academy of Sciences.

[10]  Lutz Bornmann,et al.  A multilevel modelling approach to investigating the predictive validity of editorial decisions: do the editors of a high profile journal select manuscripts that are highly cited after publication? , 2011 .

[11]  Jesper W. Schneider,et al.  Caveats for using statistical significance tests in research assessments , 2011, J. Informetrics.

[12]  Tindaro Cicero,et al.  The dispersion of research performance within and between universities as a potential indicator of the competitive intensity in higher education systems , 2012, J. Informetrics.

[13]  Tindaro Cicero,et al.  What is the appropriate length of the publication period over which to assess research performance? , 2012, Scientometrics.

[14]  Göran Falkman,et al.  Modeling uncertainty in bibliometrics and information retrieval: an information fusion approach , 2014, Scientometrics.

[15]  Marie-Laure Bougnol,et al.  Technical pitfalls in university rankings , 2015 .

[16]  H. Kiene,et al.  Bias in meta-analysis , 2006, Homeopathy.

[17]  N. Avkiran.  INVESTIGATING TECHNICAL AND SCALE EFFICIENCIES OF AUSTRALIAN UNIVERSITIES THROUGH DATA ENVELOPMENT ANALYSIS , 2001 .

[18]  Nian Cai Liu,et al.  The Academic Ranking of World Universities. , 2005 .

[19]  Per Ahlgren,et al.  The effects and their stability of field normalization baseline on relative performance with respect to citation impact: A case study of 20 natural science departments , 2011, J. Informetrics.

[20]  G. Smith,et al.  Bias in meta-analysis detected by a simple, graphical test , 1997, BMJ.

[21]  Greg Ridgeway,et al.  Latent Variable Analysis: A New Approach to University Ranking , 2005 .

[22]  D. Cox,et al.  An Analysis of Transformations , 1964 .

[23]  Lutz Bornmann,et al.  Multilevel-statistical reformulation of citation-based university rankings: The Leiden ranking 2011/2012 , 2013, J. Assoc. Inf. Sci. Technol..

[24]  Giovanni Abramo,et al.  How do you define and measure research productivity? , 2014, Scientometrics.

[25]  S. Rijcke,et al.  Bibliometrics: The Leiden Manifesto for research metrics , 2015, Nature.

[26]  W. Glänzel.  Seven Myths in Bibliometrics About facts and fiction in quantitative science studies , 2008 .

[27]  Giovanni Abramo,et al.  A national-scale cross-time analysis of university research performance , 2011, Scientometrics.

[28]  Francesca Ieva,et al.  Detecting and visualizing outliers in provider profiling via funnel plots and mixed effect models , 2014, Health Care Management Science.

[29]  Tindaro Cicero,et al.  Revisiting the scaling of citations for research assessment , 2012, J. Informetrics.

[30]  G. Cumming.  Understanding the New Statistics: Effect Sizes, Confidence Intervals, and Meta-Analysis , 2011 .

[31]  David J Spiegelhalter,et al.  Funnel plots for comparing institutional performance , 2005, Statistics in medicine.

[32]  Thed N. van Leeuwen,et al.  The Leiden ranking 2011/2012: Data collection, indicators, and interpretation , 2012, J. Assoc. Inf. Sci. Technol..

[33]  Cristiano Giuffrida,et al.  A heuristic approach to author name disambiguation in bibliometrics databases for large-scale research assessments , 2011, J. Assoc. Inf. Sci. Technol..

[34]  Tindaro Cicero,et al.  Assessing the varying level of impact measurement accuracy as a function of the citation window length , 2011, J. Informetrics.

[35]  Ying Cheng,et al.  Comparison of the effect of mean-based method and z-score for field normalization of citations at the level of Web of Science subject categories , 2014, Scientometrics.