A large-scale validation of the relationship between cross-disciplinary research and its uptake in policy-related documents, using the novel Overton altmetrics database

Abstract Cross-disciplinary research (multi-/interdisciplinarity) is incentivized by funding agencies to foster research outcomes addressing complex societal challenges. This study focuses on the link between cross-disciplinary research and its uptake in a broad set of policy-related documents. Using the new policy-oriented database Overton, matched to Scopus, logistic regression was used in assessing this relationship in publications from FP7- and H2020-supported projects. Cross-disciplinary research was captured through two lenses at the paper level, namely from the disciplinary diversity of contributing authors (DDA) and of cited references (DDR). DDA increased the likelihood that publications were cited in policy documents, with DDR possibly making a contribution, but only when publications result from the work of few authors. Citations to publications captured by Overton were found to originate in scientific advice documents, rather than in legislative or executive records. Our approach enables testing in a general way the assumption underlying many funding programs, namely that cross-disciplinary research will increase the policy relevance of research outcomes. Findings suggest that research assessments could benefit from measuring uptake in policy-related literature, following additional characterization of the Overton database; of the science-policy interactions it captures; and of the contribution of these interactions within the larger policymaking process.

[1]  Szomszor Martin,et al.  Digital Research Report: Interdisciplinary Research - Methodologies for Identification and Assessment , 2016 .

[2]  Mike Thelwall,et al.  Web indicators for research evaluation. Part 1: Citations and links to academic articles from the Web , 2015 .

[3]  Ismael Rafols,et al.  On the relationship between interdisciplinarity and impact: different modalities of interdisciplinarity lead to different types of impact , 2014, ArXiv.

[4]  Éric Archambault,et al.  Article-level classification of scientific publications: A comparison of deep learning, direct citation and bibliographic coupling , 2021, PloS one.

[5]  Elisabeth Lex,et al.  Next-generation altmetrics: responsible metrics and evaluation for open science , 2016 .

[6]  Michael Thelwall,et al.  Web Indicators for Research Evaluation: A Practical Guide , 2016, Synthesis Lectures on Information Concepts, Retrieval, and Services.

[7]  S. V. D. Hel,et al.  New science for global sustainability? The institutionalisation of knowledge co-production in Future Earth , 2016 .

[8]  Alan L. Porter,et al.  Research Coordination Networks: Evidence of the Relationship between Funded Interdisciplinary Networking and Scholarly Impact , 2012 .

[9]  Ying Huang,et al.  Interdisciplinarity and collaboration: on the relationship between disciplinary diversity in departmental affiliations and reference lists , 2018, Scientometrics.

[10]  Jian Wang,et al.  Interdisciplinarity and Impact: Distinct Effects of Variety, Balance, and Disparity , 2014, ISSI.

[11]  Russell T. Shinohara,et al.  The emergent integrated network structure of scientific research , 2018, PloS one.

[12]  Edwin Horlings,et al.  How accurately does output reflect the nature and design of transdisciplinary research programmes , 2015 .

[13]  Qualitative Evaluation of completed Projects funded by the European Research Council 2018 , 2017 .

[14]  Rick Rylance,et al.  Grant giving: Global funders to focus on interdisciplinarity , 2015, Nature.

[15]  Kang Zhao,et al.  The more multidisciplinary the better? - The prevalence and interdisciplinarity of research collaborations in multidisciplinary institutions , 2018, J. Informetrics.

[16]  Lutz Bornmann,et al.  Policy documents as sources for measuring societal impact: how often is climate change research mentioned in policy-related documents? , 2015, Scientometrics.

[17]  Laura R. Meagher,et al.  A rose by any other name? Transdisciplinarity in the context of UK research policy , 2015 .

[18]  M. Hird,et al.  How complex international partnerships shape domestic research clusters: Difference-in-difference network formation and research re-orientation in the MIT Portugal Program , 2017 .

[19]  Lutz Bornmann,et al.  How many scientific papers are mentioned in policy-related documents? An empirical investigation using Web of Science and Altmetric data , 2016, Scientometrics.

[20]  K. Warner,et al.  The impact of tobacco control research on policy: 20 years of progress , 2012, Tobacco Control.

[21]  Luke Georghiou,et al.  Policy Paper by the Research, Innovation, and Science Policy Experts (RISE) , 2015 .

[22]  J. Houghton,et al.  Where is the evidence? Realising the value of grey literature for public policy & practice: a discussion paper , 2014 .

[23]  Brooke Struck,et al.  Reliability of Scopus author identifiers (AUIDs) for research evaluation purposes at different scales , 2019, ISSI.

[24]  L. Bromham,et al.  Interdisciplinary research has consistently lower funding success , 2016, Nature.

[25]  Mike Thelwall,et al.  The metric tide: report of the independent review of the role of metrics in research assessment and management , 2015 .

[26]  N. Vermeulen,et al.  Edinburgh Research Explorer Systems biology, interdisciplinarity and disciplinary identity , 2013 .

[27]  K. Fowle,et al.  The program and policy change framework: A new tool to measure research use in low- and middle-income countries , 2020, Research Evaluation.

[28]  Ismael Rafols,et al.  Diversity and network coherence as indicators of interdisciplinarity: case studies in bionanoscience , 2009, Scientometrics.

[29]  Stefanie Haustein,et al.  Grand challenges in altmetrics: heterogeneity, data quality and dependencies , 2016, Scientometrics.

[30]  Ismael Rafols,et al.  Interdisciplinarity and Research on Local Issues: Evidence from a Developing Country , 2013, 1304.6742.

[31]  A. Bauman,et al.  Does citation matter? Research citation in policy documents as an indicator of research impact – an Australian obesity policy case-study , 2018, Health Research Policy and Systems.

[32]  I. Ràfols,et al.  Do synthesis centers synthesize? A semantic analysis of topical diversity in research , 2021, Research policy.

[33]  Éric Archambault,et al.  Application of an “interdisciplinarity” metric at the paper level and its use in a comparative analysis of the most publishing ERA and non-ERA universities , 2016 .

[34]  L. Bornmann,et al.  Altmetrics and societal impact measurements: Match or mismatch? A literature review , 2020, El Profesional de la Información.

[35]  Keisuke Okamura,et al.  Interdisciplinarity revisited: evidence for research impact and dynamism , 2019, Palgrave Communications.

[36]  Ramon Flecha,et al.  Social impact in social media: A new method to evaluate the social impact of research , 2018, PloS one.

[37]  Brooke Struck,et al.  Impact of multidisciplinary research on innovation , 2017, ISSI.

[38]  Giovanni Abramo,et al.  A comparison of two approaches for measuring interdisciplinary research output: The disciplinary diversity of authors vs the disciplinary diversity of the reference list , 2018, J. Informetrics.

[39]  Ismael Rafols,et al.  Is science becoming more interdisciplinary? Measuring and mapping six research fields over time , 2009, Scientometrics.

[40]  S. Wutzke,et al.  Evaluating the impact of applied prevention research centres: results from a modified Delphi approach , 2017 .

[41]  A. Stirling.  A general framework for analysing diversity in science, technology and society , 2007, Journal of The Royal Society Interface.

[42]  Amanda Lawrence,et al.  Influence seekers: The production of grey literature for policy and practice , 2018, Inf. Serv. Use.

[43]  Christopher Carroll,et al.  What Can Altmetric.com Tell Us About Policy Citations of Research? An Analysis of Altmetric.com Data for Research Articles from the University of Sheffield , 2018, Front. Res. Metr. Anal..

[44]  Andy Neely,et al.  Inventors and Entrepreneurs in Academia: What Types of Skills and Experience Matter? , 2011 .

[45]  B. Belcher,et al.  Understanding and evaluating the impact of integrated problem-oriented research programmes: Concepts and considerations , 2020, Research Evaluation.

[46]  Loet Leydesdorff,et al.  Interdisciplinarity as Diversity in Citation Patterns among Journals: Rao-Stirling Diversity, Relative Variety, and the Gini coefficient , 2018, J. Informetrics.

[47]  C. Weiss.  The many meanings of research utilization. , 1979 .

[48]  R. Brownson,et al.  Understanding evidence-based public health policy. , 2009, American journal of public health.

[49]  Alan L. Porter,et al.  Assessing research network and disciplinary engagement changes induced by an NSF program , 2012 .

[50]  Éric Archambault,et al.  Towards a Multilingual, Comprehensive and Open Scientific Journal Ontology , 2013 .

[51]  T. Tribaldos,et al.  Research funding programmes aiming for societal transformations: Ten key stages , 2019, Science and Public Policy.

[52]  G. Buenstorf,et al.  Interrelated funding streams in a multi-funder university system: Evidence from the German Exzellenzinitiative , 2020 .

[53]  Cinzia Daraio,et al.  A comparison of three multidisciplinarity indices based on the diversity of Scopus subject areas of authors’ documents, their bibliography and their citing papers , 2020, Scientometrics.

[54]  Lili Wang,et al.  Knowledge flows from public science to industrial technologies , 2019, The Journal of Technology Transfer.

[55]  Laura Meagher,et al.  The role of funding agencies in creating interdisciplinary knowledge , 2013 .