Learning from the UK’s research impact assessment exercise: a case study of a retrospective impact assessment exercise and questions for the future

National governments spend significant amounts of money supporting public research. However, in an era where the international economic climate has led to budget cuts, policymakers increasingly are looking to justify the returns from public investments, including in science and innovation. The so-called ‘impact agenda’ which has emerged in many countries around the world is part of this response; an attempt to understand and articulate for the public what benefits arise from the research that is funded. The United Kingdom is the most progressed in implementing this agenda and in 2014 the national research assessment exercise, the Research Excellence Framework, for the first time included the assessment of research impact as a component. For the first time within a dual funding system, funding would be awarded not only on the basis of the academic quality of research, but also on the wider impacts of that research. In this paper we outline the context and approach taken by the UK government, along with some of the core challenges that exist in implementing such an exercise. We then synthesise, together for the first time, the results of the only two national evaluations of the exercise and offer reflections for future exercises both in the UK and internationally.

[1]  V. Bush Science, the Endless Frontier , 1999, Science, the Endless Frontier.

[2]  Jonathan Grant,et al.  Making the Grade: Methodologies for Assessing and Evidencing Research Impact , 2013 .

[3]  B. V. D. Meulen,et al.  Social Impact Assessment Methods for research and funding instruments through the study of Productive Interactions between science and society , 2011 .

[4]  S. Wooding,et al.  An analysis of the impact of research supported by the UK National Institute of Health Research , 2016 .

[5]  Catriona Manville,et al.  Assessing Research Impact: An international review of the Excellence in Innovation for Australia Trial , 2013 .

[6]  Steven A. Hill Assessing (for) impact: future assessment of the societal impact of research , 2016, Palgrave Communications.

[7]  Steve Hanney,et al.  How Can Payback from Health Services Research Be Assessed? , 1996, Journal of health services research & policy.

[8]  J. Spaapen,et al.  Introducing ‘productive interactions’ in social impact assessment , 2011 .

[9]  Nathan Rosenberg,et al.  An Overview of Innovation , 2009 .

[10]  Bengt-Åke Lundvall,et al.  National Systems of Innovation: towards a theory of innovation and interactive learning London: Pint , 1995 .

[11]  Steven Wooding,et al.  Capturing Research Impacts , 2010 .

[12]  Catriona Manville,et al.  The National Institute for Health Research at 10 Years: An Impact Synthesis: 100 Impact Case Studies. , 2016, Rand health quarterly.

[13]  Christopher Freeman,et al.  Systems of Innovation: Selected Essays in Evolutionary Economics , 2008 .

[14]  J. King A review of bibliometric and other science indicators and their role in research evaluation , 1987, J. Inf. Sci..

[15]  S. Wooding,et al.  The answer is 17 years, what is the question: understanding time lags in translational research , 2011, Journal of the Royal Society of Medicine.

[16]  Jonathan Grant,et al.  Measuring research: A guide to research evaluation frameworks and tools , 2013 .

[17]  Teun Bousema,et al.  Gametocyte carriage in uncomplicated Plasmodium falciparum malaria following treatment with artemisinin combination therapy: a systematic review and meta-analysis of individual patient data , 2016, BMC Medicine.

[18]  E. Andersen,et al.  National Systems of Innovation: Towards a Theory of Innovation and Interactive Learning , 1992 .

[19]  Trisha Greenhalgh,et al.  Research impact: a narrative review , 2016, BMC Medicine.

[20]  Stephen Hanney,et al.  Medical Research: What's It Worth? Estimating the Economic Benefits from Medical Research in the UK , 2009 .

[21]  Annette Boaz,et al.  Assessing the impact of research on policy: a literature review , 2009 .

[22]  Catriona Manville,et al.  Assessing impact submissions for REF 2014: An evaluation , 2015 .

[23]  Catriona Manville,et al.  Preparing impact submissions for REF 2014: An evaluation: Findings and observations , 2015 .

[24]  J. Molas-Gallart,et al.  Tracing productive interactions to identify social impacts: an example from the social sciences , 2011 .

[25]  Linda Butler,et al.  Capturing Research Impacts: A Review of International Practice. Documented Briefing. , 2010 .

[26]  Alis Oancea,et al.  Assessing research impact in academic clinical medicine: a study using Research Excellence Framework pilot impact indicators , 2012, BMC Health Services Research.

[27]  L. Bromham,et al.  Interdisciplinary research has consistently lower funding success , 2016, Nature.

[28]  R. Scoble,et al.  Assessment, evaluations, and definitions of research impact: A review , 2014 .

[29]  Alberto Revelli,et al.  Cancer and fertility preservation: international recommendations from an expert meeting , 2016, BMC Medicine.

[30]  A. Pollitt,et al.  A 'DECISIVE' Approach to Research Funding: Lessons from Three Retrosight Studies. , 2016, Rand health quarterly.

[31]  R. May The Scientific Wealth of Nations , 1997, Science.

[32]  Stephen Hanney,et al.  Estimating the returns to UK publicly funded cancer-related research in terms of the net value of improved health outcomes , 2014, BMC Medicine.

[33]  D. Hicks Performance-based university research funding systems , 2012 .

[34]  Ronald Rousseau,et al.  The scientific wealth of European nations: Taking effectiveness into account , 1998, Scientometrics.

[35]  M. Buxton,et al.  Estimating the economic value to societies of the impact of health research: a critical review. , 2004, Bulletin of the World Health Organization.

[36]  Bradley M. Hemminger,et al.  Altmetrics in the wild: Using social media to explore scholarly impact , 2012, ArXiv.

[37]  Albert Bravo-Biosca,et al.  The Innovation Index-Measuring the UK’s Investment Innovation and its Effects , 2012 .

[38]  P. Burridge,et al.  Quantifying the economic impact of government and charity funding of medical research on private research and development funding in the United Kingdom , 2016, BMC Medicine.

[39]  C. Weiss The many meanings of research utilization. , 1979 .

[40]  Steven Wooding,et al.  How long does biomedical research take? Studying the time taken between biomedical and health research and its translation into products, policy, and practice , 2015, Health Research Policy and Systems.

[41]  B. Martin,et al.  Assessing Basic Research : Some Partial Indicators of Scientific Progress in Radio Astronomy : Research Policy , 1987 .

[42]  Catriona Manville,et al.  Preparing Impact Submissions for REF 2014: An Evaluation. Approach and Evidence. , 2015 .