Overton: A bibliometric database of policy document citations

This paper presents an analysis of the Overton policy document database, describing the makeup of materials indexed and the nature in which they cite academic literature. We report on various aspects of the data, including growth, geographic spread, language representation, the range of policy source types included, and the availability of citation links in documents. Longitudinal analysis over established journal category schemes is used to reveal the scale and disciplinary focus of citations and determine the feasibility of developing field-normalized citation indicators. We examine how well self-reported funding outcomes collected by UK funders corresponds to data indexed in the Overton database, and if peer-review assessment of impact as measured by the UK Research Excellence Framework (REF) 2014 correlates with derived citation metrics. Our findings show that for some research topics, such as health, economics, social care and the environment, Overton contains a core set of policy documents with sufficient citation linkage to academic literature to support various citation analysis that may be informative in research evaluation, impact assessment, and policy review. The data indexed in Overton agrees with that collected via self-reporting of funding outcomes, and correlates with peer-review assessment of impact in some disciplines.

[1]  Henk F. Moed,et al.  Citation Analysis in Research Evaluation , 1899 .

[2]  Dag W. Aksnes,et al.  A Criteria-based Assessment of the Coverage of Scopus and Web of Science , 2019, J. Data Inf. Sci..

[3]  Étienne Vignola-Gagné,et al.  A large-scale validation of the relationship between cross-disciplinary research and its uptake in policy-related documents, using the novel Overton altmetrics database , 2021, Quantitative Science Studies.

[4]  Vincent A. Traag,et al.  Systematic analysis of agreement between metrics and peer review in the UK REF , 2019, Palgrave Communications.

[5]  Philipp Mayr,et al.  The journal coverage of Web of Science, Scopus and Dimensions: A comparative analysis , 2020, Scientometrics.

[6]  A. Bauman,et al.  Does citation matter? Research citation in policy documents as an indicator of research impact – an Australian obesity policy case-study , 2018, Health Research Policy and Systems.

[7]  Lutz Bornmann,et al.  Do altmetrics point to the broader impact of research? An overview of benefits and disadvantages of altmetrics , 2014, J. Informetrics.

[8]  Mike Thelwall,et al.  How is research blogged? A content analysis approach , 2015, J. Assoc. Inf. Sci. Technol..

[9]  B. Björk,et al.  How Frequently are Articles in Predatory Open Access Journals Cited , 2019, Publ..

[10]  Giovanni Abramo,et al.  The VQR, Italy's second national research assessment: Methodological failures and ranking distortions , 2015, J. Assoc. Inf. Sci. Technol..

[11]  Van Eck,et al.  CWTS Leiden Ranking 2021 , 2021 .

[12]  Mike Thelwall,et al.  Assessing the impact of disciplinary research on teaching: An automatic analysis of online syllabuses , 2008, J. Assoc. Inf. Sci. Technol..

[13]  Mike Thelwall,et al.  The discretised lognormal and hooked power law distributions for complete citation data: Best options for modelling and regression , 2016, J. Informetrics.

[14]  Qi Wang,et al.  Large-scale analysis of the accuracy of the journal classification systems of Web of Science and Scopus , 2015, J. Informetrics.

[15]  Rodrigo Costas,et al.  How well developed are altmetrics? A cross-disciplinary analysis of the presence of ‘alternative metrics’ in scientific publications , 2014, Scientometrics.

[16]  Masood Fooladi,et al.  A Comparison between Two Main Academic Literature Collections: Web of Science and Scopus Databases , 2013, ArXiv.

[17]  Robert Cottrell,et al.  Evaluating “payback” on biomedical research from papers cited in clinical guidelines: applied bibliometric study , 2000, BMJ : British Medical Journal.

[18]  M. Way,et al.  The San Francisco Declaration on Research Assessment , 2013, Journal of Cell Science.

[19]  Industrial Strategy,et al.  International comparative performance of the UK research base , 2012 .

[20]  Selçuk Beşir Demir,et al.  Scholarly databases under scrutiny , 2020, J. Libr. Inf. Sci..

[21]  Mike Thelwall,et al.  Dimensions: A Competitor to Scopus and the Web of Science? , 2018, J. Informetrics.

[22]  Rafael Aleixandre-Benavent,et al.  A systematic analysis of duplicate records in Scopus , 2015, J. Informetrics.

[23]  Matthew E Falagas,et al.  Comparison of PubMed, Scopus, Web of Science, and Google Scholar: strengths and weaknesses , 2007, FASEB journal : official publication of the Federation of American Societies for Experimental Biology.

[24]  Anne-Wil Harzing,et al.  Google Scholar, Scopus and the Web of Science: a longitudinal and cross-disciplinary comparison , 2015, Scientometrics.

[25]  Trisha Greenhalgh,et al.  Research impact: a narrative review , 2016, BMC Medicine.

[26]  Martin Szomszor,et al.  Sample size in bibliometric analysis , 2020, Scientometrics.

[27]  Christopher Carroll,et al.  What Can Altmetric.com Tell Us About Policy Citations of Research? An Analysis of Altmetric.com Data for Research Articles from the University of Sheffield , 2018, Front. Res. Metr. Anal..

[28]  Henk F. Moed,et al.  A comparative study of bibliometric past performance analysis and peer judgement , 2005, Scientometrics.

[29]  G. Lewison,et al.  The Evidence Base of International Clinical Practice Guidelines on Prostate Cancer: A Global Framework for Clinical Research Evaluation , 2020 .

[30]  Jeff Z. Pan,et al.  Resource Description Framework , 2020, Definitions.

[31]  Adèle Paul-Hus,et al.  The journal coverage of Web of Science and Scopus: a comparative analysis , 2015, Scientometrics.

[32]  Arie Rip,et al.  Evaluation of societal quality of public sector research in the Netherlands , 2000 .

[33]  Emilio Delgado López-Cózar,et al.  Google Scholar, Microsoft Academic, Scopus, Dimensions, Web of Science, and OpenCitations’ COCI: a multidisciplinary comparison of coverage via citations , 2020, Scientometrics.

[34]  D. Plenz,et al.  powerlaw: A Python Package for Analysis of Heavy-Tailed Distributions , 2013, PloS one.

[35]  L. Georghiou,et al.  Evaluation of Research: A Selection of Current Practices , 1987 .

[36]  Cassidy R. Sugimoto,et al.  Do Altmetrics Work? Twitter and Ten Other Social Web Services , 2013, PloS one.

[37]  S. Demir,et al.  Academic Incentive Allowance: Scientific Productivity, Threats, Expectations , 2018 .

[38]  D. Pendlebury.  The use and misuse of journal metrics and other citation indicators , 2009, Archivum Immunologiae et Therapiae Experimentalis.

[39]  Abraham Mendoza,et al.  Comparative Analysis of the Bibliographic Data Sources Dimensions and Scopus: An Approach at the Country and Institutional Levels , 2021, Frontiers in Research Metrics and Analytics.

[40]  G. Lewison,et al.  Lung cancer research and its citation on clinical practice guidelines. , 2021, Lung cancer.

[41]  Péter Vinkler,et al.  The case of scientometricians with the "absolute relative" impact indicator , 2012, J. Informetrics.

[42]  Duncan A. Thomas,et al.  Changing research on research evaluation: A critical literature review to revisit the agenda , 2020 .

[43]  Liz Allen,et al.  Tracking the impact of research on policy and practice: investigating the feasibility of using citations in clinical guidelines for research evaluation , 2012, BMJ Open.

[44]  Loet Leydesdorff,et al.  Caveats for the journal and field normalizations in the CWTS ("Leiden") evaluations of research performance , 2010, J. Informetrics.

[45]  Eero Lahtinen,et al.  The development of quality criteria for research: a Finnish approach. , 2005, Health promotion international.

[46]  Lutz Bornmann,et al.  What is societal impact of research and how can it be assessed? a literature survey , 2013, J. Assoc. Inf. Sci. Technol..

[47]  Xianwen Wang,et al.  An extensive analysis of the presence of altmetric data for Web of Science publications across subject fields and research topics , 2020, Scientometrics.

[48]  Joseph T. Clark.  The Impact of Science on Society , 1952 .

[49]  C. Alcaraz,et al.  Citations: results differ by database , 2012, Nature.

[50]  Steve Hanney,et al.  Evaluating the Benefits from Health Research and Development Centres , 2000 .

[51]  Ludo Waltman,et al.  Field-Normalized Citation Impact Indicators and the Choice of an Appropriate Counting Method , 2015, ISSI.

[52]  G. Sivertsen.  The Norwegian Model in Norway , 2018, J. Data Inf. Sci..

[53]  Maria Liakata,et al.  Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements , 2017, PloS one.

[54]  L. Butler,et al.  Explaining Australia’s increased share of ISI publications—the effects of a funding formula based on publication counts , 2003 .

[55]  Ludo Waltman,et al.  A review of the literature on citation impact indicators , 2015, J. Informetrics.

[56]  Vincent Larivière,et al.  Averages of ratios vs. ratios of averages: An empirical analysis of four levels of aggregation , 2011, J. Informetrics.

[57]  Michal Brzezinski,et al.  Power laws in citation distributions: evidence from Scopus , 2014, Scientometrics.

[58]  R. Kostoff.  The (scientific) wealth of nations , 2004 .

[59]  Mike Thelwall,et al.  Web citations in patents: Evidence of technological impact? , 2017, J. Assoc. Inf. Sci. Technol..

[60]  Sarah Morton,et al.  Progressing research impact assessment: A ‘contributions’ approach , 2015 .

[61]  Lutz Bornmann,et al.  Policy documents as sources for measuring societal impact: how often is climate change research mentioned in policy-related documents? , 2015, Scientometrics.

[62]  Giovanni Abramo,et al.  The effects of citation-based research evaluation schemes on self-citation behavior , 2021, J. Informetrics.

[63]  Thed N. van Leeuwen,et al.  Towards a new crown indicator: Some theoretical considerations , 2010, J. Informetrics.

[64]  Melissa C. Márquez,et al.  Science Communication in Multiple Languages Is Critical to Its Effectiveness , 2020, Frontiers in Communication.

[65]  Santo Fortunato,et al.  Characterizing and Modeling Citation Dynamics , 2011, PloS one.

[66]  Nees Jan van Eck,et al.  Evaluation of the citation matching algorithms of CWTS and iFQ in comparison to the Web of science , 2015, J. Assoc. Inf. Sci. Technol..

[67]  S. Rijcke,et al.  Bibliometrics: The Leiden Manifesto for research metrics , 2015, Nature.

[68]  Luke Georghiou.  Research evaluation in European national science and technology systems , 1995 .

[69]  Russel E. Kaufman,et al.  “Science, the Endless Frontier” , 1960, Nature.

[70]  Amalia Mas-Bleda,et al.  Estimación del valor educativo de los libros académicos que no están en inglés: el caso de España , 2018, Revista española de Documentación Científica.

[71]  Mike Thelwall,et al.  Are wikipedia citations important evidence of the impact of scholarly articles and books? , 2017, J. Assoc. Inf. Sci. Technol..

[72]  H. Moed.  CWTS crown indicator measures citation impact of a research group's publication oeuvre , 2010, J. Informetrics.

[73]  Albert N. Link,et al.  Handbook on the Theory and Practice of Program Evaluation , 2013 .

[74]  M. Szomszor,et al.  Interpreting Bibliometric Data , 2021, Frontiers in Research Metrics and Analytics.

[75]  Jonathan P. Tennant.  Web of Science and Scopus are not global databases of knowledge , 2020 .

[76]  Anke Reinhardt,et al.  Evaluation in Research and Research Funding Organisations: European Practices , 2012 .

[77]  Julia Melkers,et al.  Bibliometrics as a Tool for Research Evaluation , 2013 .

[78]  Ben R. Martin,et al.  The use of multiple indicators in the assessment of basic research , 1996, Scientometrics.

[79]  Lutz Bornmann,et al.  Do altmetrics assess societal impact in a comparable way to case studies? An empirical test of the convergent validity of altmetrics based on data from the UK research excellence framework (REF) , 2018, J. Informetrics.

[80]  Jonathan Adams,et al.  Profiling citation impact: A new methodology , 2007, Scientometrics.

[81]  A. Salter,et al.  The economic benefits of publicly funded basic research: a critical review , 2001 .

[82]  Jonathan Adams,et al.  The nature, scale and beneficiaries of research impact , 2015 .

[83]  R. Scoble,et al.  Assessment, evaluations, and definitions of research impact: A review , 2014 .

[84]  V. Larivière,et al.  Understanding the contribution of UK public health research to clinical guidelines: a bibliometric analysis , 2019, F1000Research.

[85]  Martin Szomszor,et al.  Interpreting CNCIs on a country-scale: The effect of domestic and international collaboration type , 2020, J. Informetrics.

[86]  Michael Golosovsky,et al.  Universality of citation distributions: A new understanding , 2021, Quantitative Science Studies.

[87]  Robert E. Evenson,et al.  Economic Benefits from Research: An Example from Agriculture , 1979, Science.

[88]  Ludo Waltman,et al.  Large-scale comparison of bibliographic data sources: Scopus, Web of Science, Dimensions, Crossref, and Microsoft Academic , 2020, Quantitative Science Studies.

[89]  Vincent Larivière,et al.  Modeling a century of citation distributions , 2008, J. Informetrics.

[90]  Kayvan Kousha,et al.  Web of Science and Scopus language coverage , 2019, Scientometrics.

[91]  Mike Thelwall,et al.  Google Scholar, Web of Science, and Scopus: a systematic comparison of citations in 252 subject categories , 2018, J. Informetrics.

[92]  Emilio Delgado López-Cózar,et al.  The evolution of research activity in Spain , 2003, Research Policy.

[93]  Diana Hicks.  Overview of models of performance-based research funding systems , 2010 .

[94]  Magnus Eriksson,et al.  A new database of the references on international clinical practice guidelines: a facility for the evaluation of clinical research , 2019, Scientometrics.

[95]  Rodrigo Costas,et al.  Do “altmetrics” correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective , 2014, J. Assoc. Inf. Sci. Technol..

[96]  Ismael Rafols,et al.  Under-Reporting Research Relevant to Local Needs in The Global South. Database Biases in the Representation of Knowledge on Rice , 2020, ISSI.

[97]  Lutz Bornmann,et al.  Do altmetrics correlate with the quality of papers? A large-scale empirical study based on F1000Prime data , 2017, PloS one.

[98]  D. Hicks,et al.  Reception of Spanish sociology by domestic and foreign audiences differs and has consequences for evaluation , 2015 .

[99]  John Creedy,et al.  An evaluation of metrics used by the Performance-based Research Fund process in New Zealand , 2018, New Zealand Economic Papers.

[100]  Lutz Bornmann,et al.  Alternative metrics in scientometrics: a meta-analysis of research into three altmetrics , 2014, Scientometrics.