Progressing research impact assessment: A ‘contributions’ approach

There is an increasing interest in demonstrating the outcomes from research for the purposes of learning, accountability, or to demonstrate the value of research investments. However, assessing the impact of social science research on policy and practice is challenging. The ways in which research is taken up, used, and reused in policy and practice settings means that linking research processes or outputs to wider changes is difficult, and timescales are hard to predict. This article proposes an empirically grounded framework for assessing the impact of research— the Research Contribution Framework. A case study approach was adopted to explore the nature of research impact and how it might be assessed. Findings were used to design, develop, and test a framework to assess the contribution of research to relevant areas of policy and practice and to articulate wider benefits. The framework has been adapted from contribution analysis, using the idea of ‘contribution’ to help explain the ways research is taken up and used to influence policy and practice. The framework allows for a focus on the roles of research users, and examines both processes and outcomes. It is argued that this approach gets round some of the common problems in assessing impact. It provides a method of linking research and knowledge exchange to wider outcomes whilst acknowledging and including contextual factors that help or hinder research impact. It is practical, balancing robustness with feasibility. It is adaptable for a wide range of content, types of impact assessment, and purposes.

[1]  Mywish K. Maredia,et al.  Integration of environmental impacts into ex-post assessments of international agricultural research: Conceptual issues, applications, and the way forward , 2012 .

[2]  Helen M. Smith,et al.  Interpreting Qualitative Data: Methods for Analyzing Talk, Text and Interaction 3rd Edition , 2006 .

[3]  Jon Bannister,et al.  Knowledge mobilisation and the civic academy: the nature of evidence, the roles of narrative and the potential of contribution analysis , 2013 .

[4]  J. Gabbay,et al.  Evidence based guidelines or collectively constructed “mindlines?” Ethnographic study of knowledge management in primary care , 2004, BMJ : British Medical Journal.

[5]  Martin Buxton,et al.  The payback of ‘Payback’: challenges in assessing research impact , 2011 .

[6]  Jennifer Rubin,et al.  Policy and practice impacts of research funded by the Economic and Social Research Council: A case study of the Future of Work programme, supporting data , 2007 .

[7]  Susanne MacGregor,et al.  Evidence and Policy , 2017 .

[8]  R. Scoble,et al.  Assessment, evaluations, and definitions of research impact: A review , 2014 .

[9]  Annette Boaz,et al.  Assessing the impact of research on policy: a literature review , 2009 .

[10]  Annette Boaz,et al.  Real-world approaches to assessing the impact of environmental research on policy , 2011 .

[11]  Martin Rein,et al.  Social Science And Public Policy , 1976 .

[12]  A. Best,et al.  Systems thinking, knowledge and action: towards better models and methods , 2010 .

[13]  N. Mays,et al.  Describing the impact of health research: a Research Impact Framework , 2006, BMC Health Services Research.

[14]  J. Aked Pathways to impact , 2016 .

[15]  J. Molas-Gallart,et al.  Tracing productive interactions to identify social impacts: an example from the social sciences , 2011 .

[16]  Sarah Morton,et al.  Creating research impact: the roles of research users in interactive research mobilisation , 2015 .

[17]  Masood Ahmed,et al.  Bridging research and policy , 2005, Nature Climate Change.

[18]  Pat R I C I,et al.  Rogers : Using Programme Theory to Evaluate Complicated 29 Using Programme Theory to Evaluate Complicated and Complex Aspects of Interventions , 2007 .

[19]  Carol H. Weiss,et al.  Chapter 3 The haphazard connection: Social science and public policy , 1995 .

[20]  Jennifer Rubin,et al.  Policy and practice impacts of research funded by the Economic and Social Research Council: A case study of the Future of Work programme, approach and analysis , 2007 .

[21]  Thomas Delahais,et al.  Applying contribution analysis: Lessons from five years of practice , 2012 .

[22]  Sandra Nutley Assessing Research Impact , 2009 .

[23]  Steven Wooding,et al.  Proposed methods for reviewing the outcomes of health research: the impact of funding by the UK's 'Arthritis Research Campaign' , 2004, Health research policy and systems.

[24]  Vicky Ward,et al.  Developing a Framework for Transferring Knowledge Into Action: A Thematic Analysis of the Literature , 2009, Journal of health services research & policy.

[25]  Julie K. Johnson,et al.  A practical example of Contribution Analysis to a public health intervention , 2014 .

[26]  A. V. D. Ven,et al.  Knowledge for Theory and Practice , 2006 .

[27]  C. Donovan,et al.  State of the art in assessing research impact: introduction to a special issue , 2011 .

[28]  Laura R. Meagher,et al.  Flows of knowledge, expertise and influence: a method for assessing policy and practice impacts from social science research , 2008 .

[29]  C. Donovan,et al.  The Australian Research Quality Framework: A live experiment in capturing the social, economic, environmental, and cultural returns of publicly funded research , 2008 .

[30]  Mark Pearson,et al.  Handling Qualitative Data: A Practical Guide , 2005 .

[31]  Martin J Buxton,et al.  The utilisation of health research in policy-making : concepts , examples and methods of assessment , 2003 .

[32]  Azlina Mohd. Kosnin,et al.  Assessing the non-academic aspects of children with learning disabilities , 2007 .

[33]  C. Adair,et al.  Knowledge transfer and exchange: review and synthesis of the literature. , 2007, The Milbank quarterly.

[34]  Lawrence R. Jones,et al.  Strategies for Public Management Reform , 2004 .

[35]  Elske van de Fliert,et al.  Impact pathway evaluation: an approach for achieving and attributing impact in complex systems , 2003 .

[36]  K. Eisenhardt Building theories from case study research , 1989, STUDI ORGANIZZATIVI.

[37]  D. Silverman Interpreting Qualitative Data: Methods for Analysing Talk, Text and Interaction , 1994 .

[38]  E. Crewe,et al.  Bridging Research and Policy: Context, Evidence and Links , 2002 .

[39]  Patricia J. Rogers,et al.  Using Programme Theory to Evaluate Complicated and Complex Aspects of Interventions , 2008 .

[40]  J. Spaapen,et al.  Introducing ‘productive interactions’ in social impact assessment , 2011 .

[41]  Mita Marra,et al.  Evaluating the Complex: Attribution, Contribution and Beyond , 2017 .

[42]  Jordi Molas-Gallart,et al.  Assessing the non-academic impact of grant-funded socio-economic research: results from a pilot study. , 2000 .

[43]  Sarah Morton,et al.  Assessing research impact: A case study of participatory research , 2013 .

[44]  Robert Cottrell,et al.  Evaluating “payback” on biomedical research from papers cited in clinical guidelines: applied bibliometric study , 2000, BMJ : British Medical Journal.

[45]  John Mayne,et al.  Addressing Attribution through Contribution Analysis: Using Performance Measures Sensibly , 2001, Canadian Journal of Program Evaluation.

[46]  Sebastian Lemire,et al.  Making contribution analysis work: A practical framework for handling influencing factors and alternative explanations , 2012 .

[47]  Heidi L. Chorzempa,et al.  Evaluating health research impact: Development and implementation of the Alberta Innovates – Health Solutions impact framework , 2012 .

[48]  Sandra Nutley,et al.  Evidence and Policy Networks: the UK debate about sex offender community notification , 2008 .

[49]  Peter Knight,et al.  Small-Scale Research , 2002 .

[50]  Sarah Morton Exploring and assessing social research impact : a case study of a research partnership’s impacts on policy and practice , 2012 .

[51]  Katherine E. Smith,et al.  Health inequalities in Scotland and England: the contrasting journeys of ideas from research into policy. , 2007, Social science & medicine.

[52]  Helen Butler,et al.  Knowledge theories can inform evaluation practice: What can a complexity lens add? , 2009 .

[53]  H. Davies,et al.  Using Evidence: How research can inform public services , 2007 .

[54]  John Lavis,et al.  Measuring the impact of health research , 2003, Journal of health services research & policy.

[55]  Michael Quinn Patton,et al.  Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use , 2010 .

[56]  Jennifer Rubin,et al.  Policy and practice impacts of research funded by the Economic and Social Research Council , 2007 .

[57]  David Phipps A Report Detailing the Development of a University-Based Knowledge Mobilization Unit that Enhances Research Outreach and Engagement , 2012 .

[58]  Joan M.V. Pons,et al.  Assessment of the impact of a clinical and health services research call in Catalonia , 2012 .

[59]  Yoshiko Saitoh,et al.  Bibliometric techniques in the evaluation of federally funded research in the United States , 2004 .

[60]  Stephen Hanney,et al.  Evaluating Health Research Funding in Ireland , 2011 .

[61]  John Mayne,et al.  Contribution analysis: An approach to exploring cause and effect , 2008 .

[62]  A. Bryman Social Research Methods , 2001 .

[63]  M. Rader,et al.  Social Science Research on CAD/CAM , 1988, Physica-Verlag HD.

[64]  Julius Court,et al.  Bridging research and policy in international development: an analytical and practical framework , 2006 .

[65]  Diana Hicks Bibliometric evaluation of federally funded research in the United States , 2004 .