Citations, Citation Indicators, and Research Quality: An Overview of Basic Concepts and Theories

Citations are increasingly used as performance indicators in research policy and within the research system. Usually, citations are assumed to reflect the impact of the research or its quality. What is the justification for these assumptions and how do citations relate to research quality? These and similar issues have been addressed through several decades of scientometric research. This article provides an overview of some of the main issues at stake, including theories of citation and the interpretation and validity of citations as performance measures. Research quality is a multidimensional concept, where plausibility/soundness, originality, scientific value, and societal value commonly are perceived as key characteristics. The article investigates how citations may relate to these various research quality dimensions. It is argued that citations reflect aspects related to scientific impact and relevance, although with important limitations. On the contrary, there is no evidence that citations reflect other key dimensions of research quality. Hence, an increased use of citation indicators in research evaluation and funding may imply less attention to these other research quality dimensions, such as solidity/plausibility, originality, and societal value.

[1]  P. Seglen,et al.  Citations and journal impact factors: questionable indicators of research quality , 1997, Allergy.

[2]  M. Buxton,et al.  Identifying the impact of diabetes research , 2006, Diabetic medicine : a journal of the British Diabetic Association.

[3]  G. Nigel Gilbert.  Referencing as Persuasion , 1977 .

[4]  G Lewison,et al.  The impact of cancer research: how publications influence UK cancer clinical guidelines , 2008, British Journal of Cancer.

[5]  Ben R. Martin,et al.  The use of multiple indicators in the assessment of basic research , 1996, Scientometrics.

[6]  Steve Hanney,et al.  Tracing the indirect societal impacts of biomedical research: development and piloting of a technique based on citations , 2016, Scientometrics.

[7]  Loet Leydesdorff.  Words and co-words as indicators of intellectual organization , 1989 .

[8]  Michael H. MacRoberts,et al.  Problems of citation analysis , 1996, Scientometrics.

[9]  Lutz Bornmann,et al.  What do we know about the h index? , 2007, J. Assoc. Inf. Sci. Technol..

[10]  Terttu Luukkonen,et al.  Why has Latour's theory of citations been ignored by the bibliometric community? discussion of sociological interpretations of citation analysis , 2006, Scientometrics.

[11]  J. E. Hirsch,et al.  An index to quantify an individual's scientific research output , 2005, Proc. Natl. Acad. Sci. USA.

[12]  Alexander Rushforth,et al.  Accounting for Impact? The Journal Impact Factor and the Making of Biomedical Research in the Netherlands , 2015, Minerva.

[13]  Liv Langfeldt,et al.  Assessing the broader impacts of research: A review of methods and practices , 2015 .

[14]  Peter Vinkler,et al.  The Evaluation of Research by Scientometric Indicators , 2010 .

[15]  D. Aksnes.  Citations and their use as indicators in science policy : studies of validity and applicability issues with a particular focus on highly cited papers , 2005 .

[16]  Manuel Núñez-Nickel,et al.  The multilayered nature of reference selection , 2009, J. Assoc. Inf. Sci. Technol..

[17]  T. Dick,et al.  Foreword , 2010, Respiratory Physiology & Neurobiology.

[18]  Lutz Bornmann,et al.  Diversity, value and limitations of the journal impact factor and alternative metrics , 2012, Rheumatology International.

[19]  Scandinavian Simvastatin Survival Study Group.  Randomised trial of cholesterol lowering in 4444 patients with coronary heart disease: the Scandinavian Simvastatin Survival Study (4S) , 1994, The Lancet.

[20]  Thed N. van Leeuwen,et al.  Towards a new crown indicator: Some theoretical considerations , 2010, J. Informetrics.

[21]  O. H. Lowry,et al.  Protein measurement with the Folin phenol reagent. , 1951, The Journal of biological chemistry.

[22]  Mike Thelwall,et al.  The metric tide: report of the independent review of the role of metrics in research assessment and management , 2015 .

[23]  Uno Fors,et al.  Evaluating research: A multidisciplinary approach to assessing research practice and quality , 2016 .

[24]  Loet Leydesdorff,et al.  Citations: Indicators of significance? , 1989, Scientometrics.

[25]  Björn Hellqvist,et al.  Referencing in the humanities and its implications for citation analysis , 2010, J. Assoc. Inf. Sci. Technol..

[26]  Blaise Cronin,et al.  The citation process: The role and significance of citations in scientific communication , 1984 .

[27]  C. Le Pair.  Formal evaluation methods : their utility and limitations , 1995 .

[28]  E. Garfield,et al.  Of Nobel class: A citation perspective on high impact research authors , 1992, Theoretical medicine.

[29]  Richard Van Noorden,et al.  The top 100 papers , 2014, Nature.

[30]  Rosy Jan,et al.  Citation analysis of Library Trends , 2009, Webology.

[31]  Michael H. MacRoberts,et al.  Problems of citation analysis: A critical review , 1989, JASIS.

[32]  Mengxiong Liu,et al.  Progress in Documentation the Complexities of citation Practice: a Review of citation studies , 1993, J. Documentation.

[33]  Julian Warner,et al.  A critical review of the application of citation studies to the Research Assessment Exercises , 2000, J. Inf. Sci..

[34]  Robert Cottrell,et al.  Evaluating “payback” on biomedical research from papers cited in clinical guidelines: applied bibliometric study , 2000, BMJ : British Medical Journal.

[35]  Katrin Weller.  Social Media and Altmetrics: An Overview of Current Alternative Approaches to Measuring Scholarly Impact , 2015 .

[36]  Stephen Hanney,et al.  The ‘Payback Framework’ explained , 2011 .

[37]  M. Meyer.  Does science push technology? Patents citing scientific literature , 2000 .

[38]  van Raan,et al.  Advanced bibliometric methods to assess research performance and scientific development: basic principles and recent practical applications , 1993 .

[39]  Terttu Luukkonen,et al.  Conservatism and risk-taking in peer review: Emerging ERC practices , 2012 .

[40]  B. Martin,et al.  Some partial indicators of scientific progress in radio astronomy , 1983 .

[41]  Lutz Bornmann,et al.  On the causes of subject-specific citation rates in Web of Science , 2014, Scientometrics.

[42]  Ed J. Rinia,et al.  COMPARATIVE ANALYSIS OF A SET OF BIBLIOMETRIC INDICATORS AND CENTRAL PEER REVIEW CRITERIA. EVALUATION OF CONDENSED MATTER PHYSICS IN THE NETHERLANDS , 1998 .

[43]  D. Aksnes.  CHARACTERISTICS OF HIGHLY CITED PAPERS , 2003 .

[44]  Tibor Braun,et al.  Relative indicators and relational charts for comparative assessment of publication output and citation impact , 1986, Scientometrics.

[45]  John P. A. Ioannidis,et al.  Citation Metrics: A Primer on How (Not) to Normalize , 2016, PLoS biology.

[46]  Ludo Waltman,et al.  Source normalized indicators of citation impact: an overview of different approaches and an empirical comparison , 2012, Scientometrics.

[47]  J. Li,et al.  Triumph of the Heart: The Story of Statins , 2009 .

[48]  Lutz Bornmann,et al.  What is societal impact of research and how can it be assessed? a literature survey , 2013, J. Assoc. Inf. Sci. Technol..

[49]  Christian Catalini,et al.  The incidence and role of negative citations in science , 2015, Proceedings of the National Academy of Sciences.

[50]  F. Fidler,et al.  Are Psychology Journals Anti-replication? A Snapshot of Editorial Practices , 2017, Front. Psychol..

[51]  Peter Willett,et al.  Readers' perceptions of authors' citation behaviour , 2013, J. Documentation.

[52]  Evaristo Jiménez-Contreras,et al.  Reviewers’ Ratings and Bibliometric Indicators: Hand in Hand When Assessing Over Research Proposals? , 2013, PloS one.

[53]  Loet Leydesdroff.  Words and co-words as indicators of intellectual organization , 1989 .

[54]  S. Rijcke,et al.  Bibliometrics: The Leiden Manifesto for research metrics , 2015, Nature.

[55]  D. Aksnes,et al.  Peer reviews and bibliometric indicators: a comparative study at a Norwegian university , 2004 .

[56]  G. Holden,et al.  Bibliometrics , 2005, Social work in health care.

[57]  Paul Wouters,et al.  Beyond the holy grail: From citation theory to indicator theories , 1999, Scientometrics.

[58]  Liv Langfeldt,et al.  How Professors Think: Inside the Curious World of Academic Judgment , 2011 .

[59]  R. Cagan.  The San Francisco Declaration on Research Assessment , 2013, Disease Models & Mechanisms.

[60]  Tindaro Cicero,et al.  Evaluating scientific research in Italy: The 2004–10 research evaluation exercise , 2015 .

[61]  D J PRICE,et al.  NETWORKS OF SCIENTIFIC PAPERS. , 1965, Science.

[62]  Henk F. Moed,et al.  Citation Analysis in Research Evaluation , 1899 .

[63]  E. Garfield,et al.  Can Citation Indexing Be Automated ? , 1964 .

[64]  Susan E. Cozzens,et al.  What do citations count? the rhetoric-first model , 1989, Scientometrics.

[65]  M. Polanyi.  The Republic of science , 1962 .

[66]  M. Polanyi.  The Republic of Science: Its Political and Economic Theory , 1962 .

[67]  Francisco Herrera,et al.  h-Index: A review focused in its variants, computation and standardization for different scientific fields , 2009, J. Informetrics.

[68]  Jeppe Nicolaisen,et al.  Citation analysis , 2007, Annu. Rev. Inf. Sci. Technol..

[69]  Henry Voos,et al.  Are All Citations Equal? Or, Did We Op. Cit. Your Idem?. , 1976 .

[70]  D. Fanelli.  How Many Scientists Fabricate and Falsify Research? A Systematic Review and Meta-Analysis of Survey Data , 2009, PloS one.

[71]  Caroline S. Wagner,et al.  Do Nobel Laureates Create Prize-Winning Networks? An Analysis of Collaborative Research in Physiology or Medicine , 2015, PloS one.

[72]  Mike Thelwall,et al.  A combined bibliometric indicator to predict article impact , 2011, Inf. Process. Manag..

[73]  Giovanni Abramo,et al.  Evaluating research: from informed peer review to bibliometrics , 2011, Scientometrics.

[74]  Paul Wouters,et al.  Evaluation practices and effects of indicator use : a literature review , 2016 .

[75]  Tim C. E. Engels,et al.  The representation of the social sciences and humanities in the Web of Science—a comparison of publication patterns and incentive structures in Flanders and Norway (2005–9) , 2012 .

[76]  Ludo Waltman,et al.  Counting publications and citations: Is more always better? , 2013, J. Informetrics.

[77]  Giuseppe De Nicolao,et al.  Do they agree? Bibliometric evaluation versus informed peer review in the Italian research assessment exercise , 2015, Scientometrics.

[78]  R. Merton.  The Matthew Effect in Science , 1968, Science.

[79]  Giovanni Abramo,et al.  A farewell to the MNCS and like size-independent indicators , 2016, J. Informetrics.

[80]  Lutz Bornmann,et al.  Measuring the societal impact of research , 2012, EMBO reports.

[81]  Cassidy R. Sugimoto,et al.  Bias in peer review , 2013, J. Assoc. Inf. Sci. Technol..

[82]  Wolfgang Glänzel,et al.  Subject field characteristic citation scores and scales for assessing research performance , 1987, Scientometrics.

[83]  Eugene Garfield,et al.  Validation of citation analysis , 1997 .

[84]  Adam Eyre-Walker,et al.  The Assessment of Science: The Relative Merits of Post-Publication Review, the Impact Factor, and the Number of Citations , 2013, PLoS biology.

[85]  Ludo Waltman,et al.  A review of the literature on citation impact indicators , 2015, J. Informetrics.

[86]  Anthony F.J. van Raan,et al.  Patent Citations Analysis and Its Value in Research Evaluation: A Review and a New Approach to Map Technology-relevant Research , 2017, J. Data Inf. Sci..

[87]  Henry Small,et al.  Cited Documents as Concept Symbols , 1978 .

[88]  Brian D. Earp,et al.  A tragedy of the (academic) commons: interpreting the replication crisis in psychology as a social dilemma for early-career researchers , 2015, Front. Psychol..

[89]  D. Aksnes,et al.  Researchers’ perceptions of citations , 2009 .

[90]  Jochen Gläser,et al.  Integrating Scientometric Indicators into Sociological Studies: Methodical and Methodological Problems , 2001, Scientometrics.

[91]  Thed N. van Leeuwen,et al.  Towards a new crown indicator: an empirical analysis , 2010, Scientometrics.

[92]  Anthony F. J. van Raan.  Comparison of the Hirsch-index with standard bibliometric indicators and with peer judgment for 147 chemistry research groups , 2013, Scientometrics.

[93]  Fredrik Niclas Piro,et al.  How can differences in international university rankings be explained? , 2016, Scientometrics.

[94]  Charles Oppenheim,et al.  The correlation between citation counts and the 1992 research assessment exercise ratings for British research in genetics, anatomy and archaeology , 1997, J. Documentation.

[95]  Vincent A. Traag,et al.  Systematic analysis of agreement between metrics and peer review in the UK REF , 2019, Palgrave Communications.

[96]  J. E. Pedersen,et al.  Inhalation of levomepromazine in severe acute asthma , 1994, The Lancet.

[97]  Michael Bryce,et al.  Test 5.14.4. Deposit 18 June 15:43, embargoed 18/07/2019 : Article -> Review article , 2019 .

[98]  Yves Gingras,et al.  Why it has become more difficult to predict Nobel Prize winners: a bibliometric analysis of nominees and winners of the chemistry and physics prizes (1901–2007) , 2009, Scientometrics.

[99]  Plergiorgio Strata,et al.  Citation analysis , 1995, Nature.

[100]  A. V. van Raan,et al.  Patent Citations Analysis and Its Value in Research Evaluation: A Review and a New Approach to Map Technology-relevant Research , 2017 .

[101]  G. Péteri.  Evaluating science and scientists: An East‐West dialogue on research evaluation in post‐communist Europe , 2001 .

[102]  A. Casadevall,et al.  Misconduct accounts for the majority of retracted scientific publications , 2012, Proceedings of the National Academy of Sciences.

[103]  G. Gilbert.  Referencing as Persuasion , 1977 .

[104]  Eugene Garfield,et al.  Is citation analysis a legitimate evaluation tool? , 2005, Scientometrics.

[105]  Stephen Cole,et al.  Social Stratification in Science , 1974 .

[106]  Lutz Bornmann,et al.  Scientific peer review , 2011, Annu. Rev. Inf. Sci. Technol..

[107]  B. Frey,et al.  Ranking Games , 2015, Evaluation review.

[108]  Loet Leydesdorff,et al.  Caveats for the journal and field normalizations in the CWTS ("Leiden") evaluations of research performance , 2010, J. Informetrics.

[109]  Peter Weingart,et al.  Impact of bibliometrics upon the science system: Inadvertent consequences? , 2005, Scientometrics.

[110]  J. Law.  The Heterogeneity of Texts , 1986 .

[111]  S. Cole.  Making Science: Between Nature and Society , 1992 .

[112]  Loet Leydesdorff,et al.  Professional and citizen bibliometrics: complementarities and ambivalences in the development and use of indicators—a state-of-the-art report , 2016, Scientometrics.

[113]  V. Durieux,et al.  Bibliometric indicators: quality measurements of scientific publication. , 2010, Radiology.

[114]  Anthony F. J. van Raan,et al.  Advanced bibliometric methods as quantitative core of peer review based evaluation and foresight exercises , 1996, Scientometrics.

[115]  D. Chubin,et al.  Peerless Science: Peer Review and U. S. Science Policy , 1990 .

[116]  Carol Tenopir,et al.  The role of trust and authority in the citation behaviour of researchers , 2015, Inf. Res..

[117]  Gunnar Sivertsen,et al.  Excellence and growth dynamics: A comparative study of the Matthew effect , 2015 .

[118]  A. Raan.  Measuring Science: Capita Selecta of Current Main Issues , 2004 .

[119]  Carole J. Lee.  Commensuration Bias in Peer Review , 2015, Philosophy of Science.

[120]  Lutz Bornmann,et al.  Further steps towards an ideal method of measuring citation performance: The avoidance of citation (ratio) averages in field-normalization , 2011, J. Informetrics.

[121]  Ludo Waltman,et al.  On the calculation of percentile-based bibliometric indicators , 2012, J. Assoc. Inf. Sci. Technol..

[122]  P. Seglen,et al.  From bad to worse: evaluation by Journal Impact. , 1989, Trends in biochemical sciences.

[123]  P. Seglen,et al.  Citation rates and journal impact factors are not suitable for evaluation of research. , 1998, Acta orthopaedica Scandinavica.

[124]  Andrea Bergmann,et al.  Citation Indexing Its Theory And Application In Science Technology And Humanities , 2016 .

[125]  Loet Leydesdorff,et al.  Group‐based trajectory modeling (GBTM) of citations in scholarly literature: Dynamic qualities of “transient” and “sticky knowledge claims” , 2013, J. Assoc. Inf. Sci. Technol..

[126]  Vincent Larivière,et al.  Averages of ratios vs. ratios of averages: An empirical analysis of four levels of aggregation , 2011, J. Informetrics.

[127]  Stefan Hornbostel,et al.  Funding of young scientist and scientific excellence , 2009, Scientometrics.

[128]  Arie Rip,et al.  Qualitative conditions of scientometrics: The new challenges , 2006, Scientometrics.

[129]  Sven E. Hug,et al.  Criteria for assessing research quality in the humanities: a Delphi study among scholars of English literature, German literature and art history , 2013 .

[130]  Ben R. Martin,et al.  The Research Excellence Framework and the ‘impact agenda’: are we creating a Frankenstein monster? , 2011 .

[131]  P. Bazeley.  Conceptualising research performance , 2010 .

[132]  Lutz Bornmann,et al.  What do citation counts measure? A review of studies on citing behavior , 2008, J. Documentation.

[133]  Anne-Wil Harzing,et al.  Google Scholar, Scopus and the Web of Science: a longitudinal and cross-disciplinary comparison , 2015, Scientometrics.

[134]  Cassidy R. Sugimoto,et al.  The Citation: From Culture to Infrastructure , 2014 .

[135]  Anthony F. J. van Raan,et al.  Theory‐changing breakthroughs in science: The impact of research teamwork on scientific discoveries , 2016, J. Assoc. Inf. Sci. Technol..