Using categorisations of citations when assessing the outcomes from health research

SummaryThis paper describes an attempt to explore how far a categorisation of citations could be used as part of an assessment of the outcomes from health research. A large-scale project to assess the outcomes from basic, or early clinical, research is being planned, but before proceeding with such a project it was thought important to test and refine the developing methods in a preliminary study. Here we describe the development, and initial application, of one element of the planned methods: an approach to categorising citations with the aim of tracing the impact made by a body of research through several generations of papers. The results from this study contribute to methodological development for the large-scale project by indicating that: only for a small minority of citing papers is the cited paper of considerable importance; the number of times a paper is cited can not be used to indicate the importance of that paper to the articles that cite it; and self-citations could play an important role in facilitating the eventual outcomes achieved from a body of research.

[1]  Ronald N. Kostoff,et al.  The use and misuse of citation analysis in research evaluation , 1998, Scientometrics.

[2]  Henry Small,et al.  Cited Documents as Concept Symbols , 1978 .

[3]  Chandra G. Prabha,et al.  Some aspects of citation behavior: A pilot study in business administration , 1983, J. Am. Soc. Inf. Sci..

[4]  Katherine W. McCain,et al.  Citation context analysis and aging patterns of journal articles in molecular genetics , 1989, Scientometrics.

[5]  E. Garfield.  When to Cite , 1996, The Library Quarterly.

[6]  J. H. Comroe,et al.  Scientific basis for the support of biomedical science. , 1976, Science.

[7]  Leo Pavičić,et al.  Citation Context Versus the Frequency Counts of Citation Histories , 1998, J. Am. Soc. Inf. Sci..

[8]  Charles Oppenheim,et al.  Highly cited old papers and the reasons why they continue to be cited , 1978, J. Am. Soc. Inf. Sci..

[9]  M. Moravcsik,et al.  Some Results on the Function and Quality of Citations , 1975 .

[10]  Richard Smith,et al.  Measuring the social impact of research , 2001, BMJ : British Medical Journal.

[11]  T. Kuhn,et al.  The Structure of Scientific Revolutions. , 1964 .

[12]  C. J. Armstrong,et al.  Information retrieval in educational technology: E.B. Duncan and R. McAleese (Editors). Aberdeen: University of Aberdeen, 1981. ISBN 0 907258 01 8. £4.50 , 1983 .

[13]  Ray McAleese,et al.  Information Retrieval in Educational Technology. Conference Proceedings of Symposium on Information Retrieval in Educational Technology (1st, Aberdeen, Scotland, April 1, 1981). , 1981 .

[14]  B. Allen,et al.  Persuasive Communities: A Longitudinal Analysis of References in the Philosophical Transactions of the Royal Society, 1665-1990 , 1994, Social studies of science.

[15]  Ronald Rousseau,et al.  The Gozinto theorem: Using citations to determine influences on a scientific publication , 1987, Scientometrics.

[16]  S. Cozzens.  Comparing the Sciences: Citation Context Analysis of Papers from Neuropharmacology and the Sociology of Science , 1985 .

[17]  Blaise Cronin,et al.  The citation process: The role and significance of citations in scientific communication , 1984 .

[18]  Terry Shinn,et al.  The Triple Helix and New Production of Knowledge , 2002 .

[19]  Terrence A. Brooks,et al.  Private acts and public objects: An investigation of citer motivations , 1985, J. Am. Soc. Inf. Sci..

[20]  Loet Leydesdorff,et al.  Theories of Citation , 1998 .

[21]  W. Shadish,et al.  Author Judgements about Works They Cite: Three Studies from Psychology Journals , 1995 .

[22]  S. Hanney,et al.  Evaluating the NHS research and development programme: will the programme give value for money? , 1998, Journal of the Royal Society of Medicine.

[23]  R Brian Haynes,et al.  Author self-citation in the diabetes literature , 2004, Canadian Medical Association Journal.

[24]  Steven Wooding,et al.  Proposed methods for reviewing the outcomes of health research: the impact of funding by the UK's 'Arthritis Research Campaign' , 2004, Health research policy and systems.

[25]  Kapseon Kim,et al.  The motivation for citing specific references by social scientists in Korea: The phenomenon of co-existing references , 2004, Scientometrics.

[26]  Kim Kapseon.  The motivation for citing specific references by social scientists in Korea: The phenomenon of co-existing references , 2004 .

[27]  I. Spiegel-Rosing.  Science Studies: Bibliometric and Content Analysis , 1977 .

[28]  Ben-Ami Lipetz,et al.  Improvement of the selectivity of citation indexes to science literature through inclusion of citation relationship indicators , 1965 .

[29]  Jochen Gläser,et al.  Integrating Scientometric Indicators into Sociological Studies: Methodical and Methodological Problems , 2001, Scientometrics.

[30]  I Chalmers,et al.  Evaluating “payback” on biomedical research , 2000, BMJ : British Medical Journal.

[31]  E. B. Duncan,et al.  Qualified Citation Indexing: Its Relevance to Educational Technology. , 1981 .

[32]  Dag W. Aksnes,et al.  A macro study of self-citation , 2003, Scientometrics.

[33]  Glänzel Wolfgang,et al.  A bibliometric approach to the role of author self-citations in scientific communication , 2004 .

[34]  V. Cano,et al.  Citation behavior: Classification, utility, and location , 1989, JASIS.

[35]  Michael J. Moravcsik,et al.  Citation patterns in scientific revolutions , 2005, Scientometrics.

[36]  M. Buxton,et al.  Routine monitoring of performance: what makes health research and development different? , 2001, Journal of health services research & policy.

[37]  J H Comroe,et al.  Ben Franklin and Open Heart Surgery , 1974, Circulation research.

[38]  B. Martin,et al.  Assessing Basic Research : Some Partial Indicators of Scientific Progress in Radio Astronomy : Research Policy , 1987 .

[39]  Susan Bonzi,et al.  Motivations for citation: A comparison of self citation and citation to others , 1991, Scientometrics.

[40]  Robert Cottrell,et al.  Evaluating “payback” on biomedical research from papers cited in clinical guidelines: applied bibliometric study , 2000, BMJ : British Medical Journal.

[41]  Denise Craghill,et al.  The impact of information research , 1987 .

[42]  Blaise Cronin.  Agreement and divergence on referencing practice , 1981 .

[43]  Daryl E. Chubin,et al.  Content Analysis of References: Adjunct or Alternative to Citation Counting? , 1975 .

[44]  Peter Vinkler,et al.  Comparative investigation of frequency and strength of motives toward referencing. The reference threshold model , 1998, Scientometrics.

[45]  M. Buxton,et al.  Payback arising from research funding: evaluation of the Arthritis Research Campaign. , 2005, Rheumatology.

[46]  G. Gilbert.  Referencing as Persuasion , 1977 .

[47]  Pit Pichappan,et al.  The other side of the coin: The intricacies of author self-citations , 2002, Scientometrics.

[48]  Steve Hanney,et al.  How Can Payback from Health Services Research Be Assessed? , 1996, Journal of health services research & policy.

[49]  Biomedical science matters for people—so its impact should be better assessed , 2002, The Lancet.

[50]  Donald Owen Case,et al.  How can we investigate citation behavior? A study of reasons for citing literature in communication , 2000, J. Am. Soc. Inf. Sci..

[51]  Jagoda Spaventi,et al.  Citation Context Versus the Frequency Counts of Citation Histories , 1998, J. Am. Soc. Inf. Sci..

[52]  Susan E. Cozzens,et al.  The knowledge pool: Measurement challenges in evaluating fundamental research programs , 1997 .

[53]  M. White,et al.  A Qualitative Study of Citing Behavior: Contributions, Criteria, and Metalevel Documentation Concerns , 1997, The Library Quarterly.

[54]  Grant Lewison.  From biomedical research to health improvement , 2004, Scientometrics.