Guideline references and academic citations as evidence of the clinical value of health research

This article introduces a new source of evidence of the value of medical‐related research: citations from clinical guidelines. These give evidence that research findings have been used to inform the day‐to‐day practice of medical staff. To identify whether citations from guidelines can give different information from that of traditional citation counts, this article assesses the extent to which references in clinical guidelines tend to be highly cited in the academic literature and highly read in Mendeley. Using evidence from the United Kingdom, references associated with the UK's National Institute of Health and Clinical Excellence (NICE) guidelines tended to be substantially more cited than comparable articles, unless they had been published in the most recent 3 years. Citation counts also seemed to be stronger indicators than Mendeley readership altmetrics. Hence, although presence in guidelines may be particularly useful to highlight the contributions of recently published articles, for older articles citation counts may already be sufficient to recognize their contributions to health in society.

[1]  Mike Thelwall,et al.  Mapping the network structure of science parks: An exploratory study of cross-sectoral interactions reflected on the web , 2012, Aslib Proc..

[2]  Rodrigo Costas,et al.  What is the impact of the publications read by the different Mendeley users? Could they help to identify alternative types of impact? PLoS ALM Workshop, San Francisco , 2013 .

[3]  A. Ford,et al.  NICE clinical guideline (CG152): the management of Crohn's disease in adults, children and young people , 2013, Alimentary pharmacology & therapeutics.

[4]  D. Clark,et al.  Implementing NICE guidelines for the psychological treatment of depression and anxiety disorders: The IAPT experience , 2011, International review of psychiatry.

[5]  Sharon Smart Clinical confidence. , 2007, Nursing standard (Royal College of Nursing (Great Britain) : 1987).

[6]  Bronwyn H Hall,et al.  Market value and patent citations , 2005 .

[7]  Mike Thelwall,et al.  Validating online reference managers for scholarly impact measurement , 2011, Scientometrics.

[8]  Tom Coupé,et al.  Science Is Golden: Academic R&D and University Patents , 2003 .

[9]  Alis Oancea,et al.  Assessing research impact in academic clinical medicine: a study using Research Excellence Framework pilot impact indicators , 2012, BMC Health Services Research.

[10]  I N Purves,et al.  Utilisation of computerised clinical guidance in general practice consultations. , 2000, Studies in health technology and informatics.

[11]  M. Trajtenberg,et al.  Universities as a Source of Commercial Technology: A Detailed Analysis of University Patenting, 19651988 , 1995, Review of Economics and Statistics.

[12]  Faiza Tabassum,et al.  Are Current UK National Institute for Health and Clinical Excellence (NICE) Obesity Risk Guidelines Useful? Cross-Sectional Associations with Cardiovascular Disease Risk Factors in a Large, Representative English Population , 2013, PloS one.

[13]  P. Malone,et al.  Will the implementation of the 2007 National Institute for Health and Clinical Excellence (NICE) guidelines on childhood urinary tract infection (UTI) in the UK miss significant urinary tract pathology? , 2012, BJU international.

[14]  S. Inouye,et al.  NICE to HELP: Operationalizing National Institute for Health and Clinical Excellence Guidelines to Improve Clinical Practice , 2014, Journal of the American Geriatrics Society.

[15]  Emily Stenke,et al.  Ulcerative colitis: management in adults, children and young people (NICE Clinical Guideline CG166) , 2014, Archives of Disease in Childhood: Education & Practice Edition.

[16]  M. Haymart,et al.  The role of clinical guidelines in patient care: thyroid hormone replacement in women of reproductive age. , 2010, Thyroid : official journal of the American Thyroid Association.

[17]  Jo Carlowe Sex, drugs and rock 'n' roll. , 2007, Nursing standard (Royal College of Nursing (Great Britain) : 1987).

[18]  G Lewison,et al.  The impact of cancer research: how publications influence UK cancer clinical guidelines , 2008, British Journal of Cancer.

[19]  Walter D. Valdivia University Start-Ups: Critical for Improving Technology Transfer , 2013 .

[20]  Mike Thelwall,et al.  When are readership counts as useful as citation counts? Scopus versus Mendeley for LIS journals , 2016, J. Assoc. Inf. Sci. Technol..

[21]  Ruben D. Restrepo,et al.  AARC Clinical Practice Guidelines: from "reference-based" to "evidence-based". , 2010, Respiratory care.

[22]  Pablo Alonso-Coello,et al.  Updated recommendations: an assessment of NICE clinical guidelines , 2014, Implementation Science.

[23]  M. Gittelman,et al.  Patent Citations as a Measure of Knowledge Flows: The Influence of Examiner Citations , 2006, The Review of Economics and Statistics.

[24]  Stephen Hanney,et al.  Estimating the returns to UK publicly funded cancer-related research in terms of the net value of improved health outcomes , 2014, BMC Medicine.

[25]  J. Grimshaw,et al.  Potential benefits, limitations, and harms of clinical guidelines , 1999, BMJ.

[26]  M. Trajtenberg A Penny for Your Quotes : Patent Citations and the Value of Innovations , 1990 .

[27]  J M Grimshaw,et al.  Effect of clinical guidelines on medical practice: a systematic review of rigorous evaluations. , 1994, Lancet.

[28]  Jens Peter Andersen,et al.  Association between quality of clinical practice guidelines and citations given to their references , 2013, ArXiv.

[29]  Vincent Larivière,et al.  Who reads research articles? An altmetrics analysis of Mendeley user categories , 2015, J. Assoc. Inf. Sci. Technol..

[30]  Liz Allen,et al.  Tracking the impact of research on policy and practice: investigating the feasibility of using citations in clinical guidelines for research evaluation , 2012, BMJ Open.

[31]  Stephen A Matthews,et al.  Clinical, sociodemographic, and service provider determinants of guideline concordant colorectal cancer care for Appalachian residents. , 2014, The Journal of rural health : official journal of the American Rural Health Association and the National Rural Health Care Association.