How to evaluate individual researchers working in the natural and life sciences meaningfully? A proposal of methods based on percentiles of citations

Although bibliometrics has been a separate research field for many years, there is still no uniformity in the way bibliometric analyses are applied to individual researchers. Therefore, this study aims to set up proposals how to evaluate individual researchers working in the natural and life sciences. 2005 saw the introduction of the h index, which gives information about a researcher’s productivity and the impact of his or her publications in a single number (h is the number of publications with at least h citations); however, it is not possible to cover the multidimensional complexity of research performance and to undertake inter-personal comparisons with this number. This study therefore includes recommendations for a set of indicators to be used for evaluating researchers. Our proposals relate to the selection of data on which an evaluation is based, the analysis of the data and the presentation of the results.

[1]  Sven Hemlin,et al.  Research on research evaluation , 1996 .

[2]  Anthony F. J. van Raan,et al.  Bibliometric statistical properties of the 100 largest European universities: prevalent scaling rules in the science system , 2007, 0704.0889.

[3]  M. Nahata,et al.  New indices in scholarship assessment. , 2009, American journal of pharmaceutical education.

[4]  H. Grupp,et al.  Indicators for national science and technology policy: how robust are composite indicators? , 2004 .

[5]  Thed N. van Leeuwen,et al.  A bibliometric classificatory approach for the study and assessment of research performance at the individual level: The effects of age on productivity and impact , 2010, J. Assoc. Inf. Sci. Technol..

[6]  Iain D. Craig,et al.  Measuring Academic Research: How to Undertake a Bibliometric Study , 2010, Learn. Publ..

[7]  Giovanni Abramo,et al.  Evaluating research: from informed peer review to bibliometrics , 2011, Scientometrics.

[8]  Cassidy R. Sugimoto,et al.  Biobibliometric profiling: An examination of multifaceted approaches to scholarship , 2012, J. Assoc. Inf. Sci. Technol..

[9]  Yuval Kalish,et al.  Making Science , 2014 .

[10]  Lutz Bornmann,et al.  A multilevel modelling approach to investigating the predictive validity of editorial decisions: do the editors of a high profile journal select manuscripts that are highly cited after publication? , 2011 .

[11]  Francisco Herrera,et al.  h-Index: A review focused in its variants, computation and standardization for different scientific fields , 2009, J. Informetrics.

[12]  Brian A. Nosek,et al.  Cumulative and Career-Stage Citation Impact of Social-Personality Psychology Programs and Their Members , 2010, Personality & social psychology bulletin.

[13]  Lutz Bornmann,et al.  A multilevel meta-analysis of studies reporting correlations between the h index and 37 different h index variants , 2011, J. Informetrics.

[14]  Rickard Danell,et al.  Can the quality of scientific work be predicted using information on the author's track record? , 2011, J. Assoc. Inf. Sci. Technol..

[15]  Lokman I. Meho,et al.  Timelines of creativity: A study of intellectual innovators in information science , 2007, J. Assoc. Inf. Sci. Technol..

[16]  Thed N. van Leeuwen,et al.  Benchmarking international scientific excellence: Are highly cited research papers an appropriate frame of reference? , 2002, Scientometrics.

[17]  Gabriel Kreiman,et al.  Nine Criteria for a Measure of Scientific Output , 2011, Front. Comput. Neurosci..

[18]  B. Jay Coleman,et al.  Benchmarking Individual Publication Productivity in Logistics , 2012 .

[19]  Péter Jacsó,et al.  Metadata mega mess in Google Scholar , 2010, Online Inf. Rev..

[20]  T. Opthof,et al.  Bibliometric data in clinical cardiology revisited. The case of 37 Dutch professors , 2011, Netherlands heart journal : monthly journal of the Netherlands Society of Cardiology and the Netherlands Heart Foundation.

[21]  Sune Lehmann,et al.  A quantitative analysis of indicators of scientific performance , 2008, Scientometrics.

[22]  Tindaro Cicero,et al.  Assessing the varying level of impact measurement accuracy as a function of the citation window length , 2011, J. Informetrics.

[23]  Thed N. van Leeuwen,et al.  A critical analysis of the journal impact factors ofAngewandte Chemie and the journal of the American Chemical Society inaccuracies in published impact factors based on overall citations only , 1996, Scientometrics.

[24]  A Nocon.  Fair assessment. , 1993, The Health service journal.

[25]  Kate E Decleene,et al.  Publication Manual of the American Psychological Association , 2011 .

[26]  Lutz Bornmann,et al.  On the problems of dealing with bibliometric data , 2014, J. Assoc. Inf. Sci. Technol..

[27]  Tibor Braun,et al.  Reference standards for citation based assessments , 2005, Scientometrics.

[28]  H. Moed.  Citation Analysis in Research Evaluation (Information Science & Knowledge Management) , 2005 .

[29]  Tibor Braun,et al.  Cross-field normalization of scientometric indicators , 1996, Scientometrics.

[30]  Leo Egghe,et al.  The Hirsch index and related impact measures , 2010, Annu. Rev. Inf. Sci. Technol..

[31]  Cristiano Giuffrida,et al.  A heuristic approach to author name disambiguation in bibliometrics databases for large-scale research assessments , 2011, J. Assoc. Inf. Sci. Technol..

[32]  Lutz Bornmann,et al.  Multiple publication on a single research study: Does it pay? The influence of number of research articles on total citation counts in biomedicine , 2007, J. Assoc. Inf. Sci. Technol..

[33]  Hariolf Grupp,et al.  Indicators for National Science and Technology Policy , 2004 .

[34]  Lutz Bornmann,et al.  What do we know about the h index? , 2007, J. Assoc. Inf. Sci. Technol..

[35]  Loet Leydesdorff,et al.  Turning the tables in citation analysis one more time: Principles for comparing sets of documents by using an “Integrated Impact Indicator” (I3) , 2011 .

[36]  Andreas Strotmann,et al.  Author name disambiguation: What difference does it make in author-based citation analysis? , 2012, J. Assoc. Inf. Sci. Technol..

[37]  Luk Arbuckle,et al.  Two h-Index Benchmarks for Evaluating the Publication Performance of Medical Informatics Researchers , 2012, Journal of medical Internet research.

[38]  N. Haslam,et al.  Quality, quantity, and impact in academic publication , 2009 .

[39]  L. Egghe,et al.  Theory and practise of the g-index , 2006, Scientometrics.

[40]  Pedro Albarrán,et al.  References made and citations received by scientific articles , 2011, J. Assoc. Inf. Sci. Technol..

[41]  L. Bornmann,et al.  The Usefulness of Peer Review for Selecting Manuscripts for Publication: A Utility Analysis Taking as an Example a High-Impact Journal , 2010, PloS one.

[42]  Dag W. Aksnes,et al.  A macro study of self-citation , 2003, Scientometrics.

[43]  Lutz Bornmann,et al.  How to analyze percentile citation impact data meaningfully in bibliometrics: The statistical analysis of distributions, percentile rank classes, and top-cited papers , 2013, J. Assoc. Inf. Sci. Technol..

[44]  Charles Oppenheim,et al.  The h-index: a broad review of a new bibliometric indicator , 2010, J. Documentation.

[45]  L. Bornmann,et al.  The state of h index research , 2009, EMBO reports.

[46]  S. Cole.  Making Science: Between Nature and Society , 1992 .

[47]  BornmannLutz,et al.  Are there better indices for evaluation purposes than the h index? A comparison of nine different variants of the h index using data from biomedicine , 2008 .

[48]  Lutz Bornmann,et al.  Diversity, value and limitations of the journal impact factor and alternative metrics , 2012, Rheumatology International.

[49]  Lutz Bornmann,et al.  OPEN PEN ACCESS CCESS , 2008 .

[50]  Linda Butler,et al.  Extending citation analysis to non-source items , 2006, Scientometrics.

[51]  Javier Ruiz-Castillo,et al.  The evaluation of citation distributions , 2011 .

[52]  Lutz Bornmann,et al.  Are there better indices for evaluation purposes than the h index? A comparison of nine different variants of the h index using data from biomedicine , 2008, J. Assoc. Inf. Sci. Technol..

[53]  Jim Taylor,et al.  The Assessment of Research Quality in UK Universities: Peer Review or Metrics? , 2011 .

[54]  Ludo Waltman,et al.  The inconsistency of the h-index , 2011, J. Assoc. Inf. Sci. Technol..

[55]  Lutz Bornmann,et al.  Distributions instead of single numbers: Percentiles and beam plots for the assessment of single researchers , 2014, J. Assoc. Inf. Sci. Technol..

[56]  Robert K. Merton,et al.  Auf den Schultern von Riesen : ein Lietfaden durch das Labyrinth der Gelehrsamkeit , 1983 .

[57]  Gabriel Pinski,et al.  Citation influence for journal aggregates of scientific publications: Theory, with application to the literature of physics , 1976, Inf. Process. Manag..

[58]  Gregory D. Webster,et al.  The research productivity of academic psychologists: assessment, trends, and best practice recommendations , 2011, Scientometrics.

[59]  J. Sahel.  Quality Versus Quantity: Assessing Individual Research Performance , 2011, Science Translational Medicine.

[60]  Péter Jacsó,et al.  Google Scholar's Ghost Authors. , 2009 .

[61]  Loet Leydesdorff,et al.  Do Scientific Advancements Lean on the Shoulders of Giants? A Bibliometric Investigation of the Ortega Hypothesis , 2010, PloS one.

[62]  Koenraad Debackere,et al.  A concise review on the role of author self-citations in information science, bibliometrics and science policy , 2006, Scientometrics.

[63]  Anthony F. J. van Raan,et al.  Measurement of Central Aspects of Scientific Research: Performance, Interdisciplinarity, Structure , 2005 .

[64]  J. C. Korevaar,et al.  Validation of bibliometric indicators in the field of mathematics , 1996, Scientometrics.

[65]  J. E. Hirsch,et al.  An index to quantify an individual's scientific research output , 2005, Proc. Natl. Acad. Sci. USA.

[66]  Tx Station.  Stata Statistical Software: Release 7. , 2001 .

[67]  Anthony F. J. van Raan,et al.  Bibliometric statistical properties of the 100 largest European research universities: Prevalent scaling rules in the science system , 2008, J. Assoc. Inf. Sci. Technol..

[68]  M. Eysenck,et al.  The correlation between RAE ratings and citation counts in psychology Technical Report , 2002 .

[69]  G. Cumming.  Understanding the New Statistics: Effect Sizes, Confidence Intervals, and Meta-Analysis , 2011 .

[70]  Lutz Bornmann,et al.  What do citation counts measure? A review of studies on citing behavior , 2008, J. Documentation.

[71]  Jian Wang,et al.  Citation time window choice for research impact evaluation , 2013, Scientometrics.

[72]  Pierre Azoulay,et al.  Incentives and Creativity: Evidence from the Academic Life Sciences , 2009 .

[73]  Henk F. Moed,et al.  Handbook of Quantitative Science and Technology Research: The Use of Publication and Patent Statistics in Studies of S&T Systems , 2004 .

[74]  Marek Kosmulski,et al.  Successful papers: A new idea in evaluation of scientific output , 2011, J. Informetrics.

[75]  D. Pendlebury.  The use and misuse of journal metrics and other citation indicators , 2009, Archivum Immunologiae et Therapiae Experimentalis.

[76]  Andreas Thor,et al.  Convergent validity of bibliometric Google Scholar data in the field of chemistry - Citation counts for papers that were accepted by Angewandte Chemie International Edition or rejected but published elsewhere, using Google Scholar, Science Citation Index, Scopus, and Chemical Abstracts , 2009, J. Informetrics.

[77]  Thed N. van Leeuwen,et al.  The Leiden ranking 2011/2012: Data collection, indicators, and interpretation , 2012, J. Assoc. Inf. Sci. Technol..

[78]  Terje Bruen Olsen,et al.  Validation of Bibliometric Indicators in the Field of Microbiology: A Norwegian Case Study , 2004, Scientometrics.

[79]  Lutz Bornmann,et al.  A better alternative to the h index , 2013, J. Informetrics.

[80]  Giovanni Abramo,et al.  Testing the trade-off between productivity and quality in research activities , 2018, J. Assoc. Inf. Sci. Technol..

[81]  Lutz Bornmann,et al.  The advantage of the use of samples in evaluative bibliometric studies , 2013, J. Informetrics.

[82]  Wolfgang Glänzel,et al.  Where demographics meets scientometrics: towards a dynamic career analysis , 2011, Scientometrics.

[83]  Anthony F. J. van Raan,et al.  Advanced bibliometric methods as quantitative core of peer review based evaluation and foresight exercises , 1996, Scientometrics.

[84]  H. Dietz,et al.  INDICATORS FOR RESEARCH PERFORMANCE EVALUATION: AN OVERVIEW , 2012, BJU international.

[85]  Fiorenzo Franceschini,et al.  The success-index: an alternative approach to the h-index for evaluating an individual’s research output , 2012, Scientometrics.

[86]  Lutz Bornmann,et al.  The problem of citation impact assessments for recent publication years in institutional evaluations , 2013, J. Informetrics.

[87]  B. Martin,et al.  Assessing Basic Research : Some Partial Indicators of Scientific Progress in Radio Astronomy : Research Policy , 1987 .

[88]  Lokman I. Meho,et al.  Ranking the research productivity of library and information science faculty and schools: An evaluation of data sources and research methods , 2005, J. Assoc. Inf. Sci. Technol..

[89]  Lutz Bornmann,et al.  Scientific peer review , 2011, Annu. Rev. Inf. Sci. Technol..

[90]  M. Lamont.  Toward a Comparative Sociology of Valuation and Evaluation , 2012 .

[91]  John Panaretos,et al.  Assessing scientific research performance and impact with single indices , 2008, Scientometrics.

[92]  魏屹东,et al.  Scientometrics , 2018, Encyclopedia of Big Data.

[93]  Loet Leydesdorff,et al.  The new Excellence Indicator in the World Report of the SCImago Institutions Rankings 2011 , 2011, J. Informetrics.

[94]  Miguel A. García-Pérez,et al.  Accuracy and completeness of publication and citation records in the Web of Science, PsycINFO, and Google Scholar: A case study for the computation of h indices in Psychology , 2010, J. Assoc. Inf. Sci. Technol..

[95]  Alan Singleton,et al.  Bibliometrics and Citation Analysis; from the Science Citation Index to Cybermetrics , 2010, Learn. Publ..

[96]  Henk F. Moed,et al.  Citation Analysis in Research Evaluation , 1899 .

[97]  A Henderson,et al.  Research productivity. , 1996, Science.

[98]  Vroni Retzer,et al.  Towards objectivity in research evaluation using bibliometric indicators – A protocol for incorporating complexity , 2009 .

[99]  Lutz Bornmann,et al.  Stata commands for importing bibliometric data and processing author address information , 2012, J. Informetrics.

[100]  R. Merton.  Priorities in scientific discovery: A chapter in the sociology of science. , 1957 .

[101]  Graham Thornicroft,et al.  Fair assessment of the merits of psychiatric research , 2007, British Journal of Psychiatry.

[102]  Henk F. Moed,et al.  The research guarantors of scientific papers and the output counting: a promising new approach , 2013, Scientometrics.

[103]  Lokman I. Meho,et al.  Ranking the research productivity of library and information science faculty and schools: An evaluation of data sources and research methods: Research Articles , 2005 .

[104]  Stephen J. Bensman.  The evaluation of research by scientometric indicators , 2011, J. Assoc. Inf. Sci. Technol..

[105]  Anthony F. J. van Raan,et al.  Fatal attraction: Conceptual and methodological problems in the ranking of universities by bibliometric methods , 2005, Scientometrics.

[106]  Henk F. Moed,et al.  The publication output and impact of academic chemistry research in the Netherlands during the 1980s: bibliometric analyses and policy implications , 1996 .

[107]  Javier Ruiz-Castillo Ucelay.  The evaluation of citation distributions , 2012 .

[108]  Kevin W Boyack,et al.  Mapping knowledge domains: Characterizing PNAS , 2004, Proceedings of the National Academy of Sciences of the United States of America.

[109]  David P. Doane,et al.  Using Beam and Fulcrum Displays to Explore Data , 2000 .

[110]  Miguel A. García-Pérez.  Accuracy and completeness of publication and citation records in the Web of Science, PsycINFO, and Google Scholar: A case study for the computation of , 2010 .