Beyond Academia – Interrogating Research Impact in the Research Excellence Framework

Big changes to the way in which research funding is allocated to UK universities were brought about in the Research Excellence Framework (REF), overseen by the Higher Education Funding Council, England. Replacing the earlier Research Assessment Exercise, the purpose of the REF was to assess the quality and reach of research in UK universities–and allocate funding accordingly. For the first time, this included an assessment of research ‘impact’, accounting for 20% of the funding allocation. In this article we use a text mining technique to investigate the interpretations of impact put forward via impact case studies in the REF process. We find that institutions have developed a diverse interpretation of impact, ranging from commercial applications to public and cultural engagement activities. These interpretations of impact vary from discipline to discipline and between institutions, with more broad-based institutions depicting a greater variety of impacts. Comparing the interpretations with the score given by REF, we found no evidence of one particular interpretation being more highly rewarded than another. Importantly, we also found a positive correlation between impact score and [overall research] quality score, suggesting that impact is not being achieved at the expense of research excellence.

[1]  Andrew Pettigrew,et al.  Scholarship with Impact , 2011 .

[2]  Magnus Sahlgren,et al.  The Word-Space Model: using distributional analysis to represent syntagmatic and paradigmatic relations between words in high-dimensional vector spaces , 2006 .

[3]  G. Gaskell,et al.  KEYWORDS in Context: Statistical Analysis of Text Features , 2000 .

[4]  Hans-Jürgen Niemann,et al.  Impact of research on development of large cooling towers , 1986 .

[5]  Jonathan Parker,et al.  The Research Excellence Framework (REF): Assessing the Impact of Social Work Research on Society , 2012 .

[6]  A. House,et al.  ‘Impact’ in the proposals for the UK's Research Excellence Framework: Shifting the boundaries of academic autonomy , 2011 .

[7]  Judith Bara,et al.  Analysing Parliamentary Debate with Computer Assistance , 2007 .

[8]  Jean-Guy Meunier,et al.  Text Mining Methods for Social Representation Analysis in Large Corpora , 2011 .

[9]  R. Phillips.  The impact agenda and geographies of curiosity , 2010 .

[10]  Catriona Manville,et al.  Assessing impact submissions for REF 2014: An evaluation , 2015 .

[11]  David Bholat,et al.  Text Mining for Central Banks , 2015 .

[12]  Annette Boaz,et al.  Real-world approaches to assessing the impact of environmental research on policy , 2011 .

[13]  G. Samuel,et al.  The Evaluation Scale: Exploring Decisions About Societal Impact in Peer Review Panels , 2016, Minerva.

[14]  T. Slater.  Impacted geographers: a response to Pain, Kesby and Askins , 2012 .

[15]  Pedro Santos,et al.  Text mining for central banks: handbook , 2015 .

[16]  Terttu Luukkonen,et al.  Conservatism and risk-taking in peer review: Emerging ERC practices , 2012 .

[17]  K. Clayton,et al.  Transactions of the Institute of British Geographers , 1959 .

[18]  Steven Wooding,et al.  UK Doubles Its “World-Leading” Research in Life Sciences and Medicine in Six Years: Testing the Claim? , 2015, PloS one.

[19]  A. Oancea.  Interpretations of Research Impact in Seven Disciplines , 2013 .

[20]  B. Francis.  Increasing Impact? An analysis of issues raised by the impact agenda in educational research , 2011, Scottish Educational Review.

[21]  Creso M. Sá,et al.  Accountability, performance assessment, and evaluation: Policy pressures and responses from research councils , 2013 .

[22]  K. Huutoniemi,et al.  Communicating and compromising on disciplinary expertise in the peer review of research proposals , 2012 .

[23]  G. Parry,et al.  The Impact of Research on the Practice of Psychotherapy , 1986 .

[24]  R. Pain,et al.  Geographies of impact: power, participation and potential , 2011 .

[25]  P. Suppes.  Impact of research on education : some case studies , 1978 .

[26]  N. Allum,et al.  Exploring public discourses about emerging technologies through statistical clustering of open-ended survey questions , 2013, Public understanding of science.

[27]  Jennifer Rubin,et al.  Assessing policy and practice impacts of social science research: the application of the Payback Framework to assess the Future of Work programme , 2011 .

[28]  STOCKHOLM UNIVERSITY , 1999 .

[29]  Martin W. Bauer,et al.  Qualitative researching with text, image and sound : a practical handbook , 2000 .

[30]  J Lomas,et al.  Finding audiences, changing beliefs: the structure of research use in Canadian health policy. , 1990, Journal of health politics, policy and law.

[31]  R. Scoble,et al.  Assessment, evaluations, and definitions of research impact: A review , 2014 .

[32]  Jonathan Adams,et al.  The nature, scale and beneficiaries of research impact , 2015 .

[33]  Mike Thelwall,et al.  The metric tide: report of the independent review of the role of metrics in research assessment and management , 2015 .

[34]  R. Watermeyer.  From Engagement to Impact? Articulating the Public Value of Academic Research , 2012 .

[35]  Ben R. Martin,et al.  The Research Excellence Framework and the ‘impact agenda’: are we creating a Frankenstein monster? , 2011 .

[36]  Catriona Manville,et al.  Preparing impact submissions for REF 2014: An evaluation: Findings and observations , 2015 .

[37]  R. Watermeyer.  Impact in the REF: issues and obstacles , 2016 .

[38]  Ellen G. Cohn,et al.  The Impact of Research on Legal Policy: The Minneapolis Domestic Violence Experiment , 1989 .