Assessing the health research’s social impact: a systematic review

In recent year, a growing attention is dedicated to the assessment of research’s social impact. While prior research has often dealt with results of research, the last decade has begun to generate knowledge on the assessment of health research’s social impact. However, this knowledge is scattered across different disciplines, research communities, and journals. Therefore, this paper analyzes the heterogeneous picture research has drawn within the past years with a focus on the health research’s social impact on different stakeholders through an interdisciplinary, systematic review. By consulting major research databases, we have analyzed 53 key journal articles bibliographically and thematically. We argued that the adoption of a multi-stakeholder could be an evolution of the existing methods used to assess impact of research. After presenting a model to assess the health research’s social impact with a multi stakeholder perspective, we suggest the implementation in the research process of three practice: a multi-stakeholder workshop on research agenda; a multi stakeholder supervisory board; a multi-stakeholder review process.

[1]  Jonathan Grant,et al.  Making the Grade: Methodologies for Assessing and Evidencing Research Impact , 2013 .

[2]  R. Frodeman,et al.  Peer review and the ex ante assessment of societal impacts , 2011 .

[3]  Lutz Bornmann,et al.  What is societal impact of research and how can it be assessed? a literature survey , 2013, J. Assoc. Inf. Sci. Technol..

[4]  C. Bradbury‐Jones,et al.  International principles of social impact assessment: lessons for research? , 2011 .

[5]  R. Scoble,et al.  Assessment, evaluations, and definitions of research impact: A review , 2014 .

[6]  Annette Boaz,et al.  Assessing the impact of research on policy: a literature review , 2009 .

[7]  Johan Bollen,et al.  Towards usage-based impact metrics: first results from the mesur project. , 2008, JCDL '08.

[8]  Ingeborg Meijer,et al.  Societal output and use of research performed by health research groups , 2010, Health research policy and systems.

[9]  J. Grimshaw,et al.  Knowledge translation of research findings , 2012, Implementation Science.

[10]  Annette Boaz,et al.  Real-world approaches to assessing the impact of environmental research on policy , 2011 .

[11]  R. Thorpe,et al.  Using Knowledge within Small and Medium-Sized Firms: A Systematic Review of the Evidence , 2005 .

[12]  Some Thoughts on Outcomes Research, Quality Improvement, and Performance Measurement , 2002, Medical care.

[13]  H. Doll,et al.  Outcome measurement in forensic mental health research: an evaluation , 2011 .

[14]  Steve Hanney,et al.  Evaluating the Benefits from Health Research and Development Centres , 2000 .

[15]  The potential impact of comparative effectiveness research on the health of minority populations. , 2010, Health affairs.

[16]  T. Lüscher,et al.  Moving from impact to influence: measurement and the changing role of medical journals. , 2012, European heart journal.

[17]  Martin J Buxton,et al.  The utilisation of health research in policy-making : concepts , examples and methods of assessment , 2003 .

[18]  A. Elzinga The New Production of Knowledge. The Dynamics of Science and Research in Contemporary Societies , 1997 .

[19]  Paul Nightingale,et al.  Peer review and the relevance gap: Ten suggestions for policy-makers , 2007 .

[20]  Sarah Morton,et al.  Progressing research impact assessment: A ‘contributions’ approach , 2015 .

[21]  Bas J Blaauboer,et al.  Evaluation of research activities and research needs to increase the impact and applicability of alternative testing strategies in risk assessment practice. , 2011, Regulatory toxicology and pharmacology : RTP.

[22]  H. V. Lente,et al.  The Mixed Blessing of Mode 2 Knowledge Production , 2010 .

[23]  Javier M. Ekboir,et al.  Why impact analysis should not be used for research evaluation and what the alternatives are , 2003 .

[24]  Ulrich Schmoch,et al.  New activities of universities in transfer and extension: multiple requirements and manifold solutions , 2009 .

[25]  Joan M.V. Pons,et al.  Assessment of the impact of a clinical and health services research call in Catalonia , 2012 .

[26]  A. Fendrick,et al.  Real-world impact of comparative effectiveness research findings on clinical practice. , 2014, The American journal of managed care.

[27]  B. Poti,et al.  Evaluation of firm R&D and innovation support: new indicators and the ex-ante prediction of ex-post additionality , 2011 .

[28]  Albert W Wu,et al.  The Use of Patient-reported Outcomes (PRO) Within Comparative Effectiveness Research: Implications for Clinical Practice and Health Care Policy , 2012, Medical care.

[29]  Loet Leydesdorff,et al.  Past performance, peer review and project selection: a case study in the social and behavioral sciences , 2009, 0911.1306.

[30]  B. Frey,et al.  Do Rankings Reflect Research Quality? , 2008, SSRN Electronic Journal.

[31]  M. Wagner,et al.  Editorial: The Challenges and Opportunities of Sustainable Development for Entrepreneurship and Small Business , 2012 .

[32]  Elizabeth M Denholm,et al.  Translational research in environmental health sciences. , 2008, Translational research : the journal of laboratory and clinical medicine.

[33]  G. Falavigna,et al.  Public Health Institutions, Clinical Research and Protection System of Patients’ Rights: An Impact Evaluation of Public Policy , 2014 .

[34]  M. Zelefsky,et al.  Evaluation of Adherence to Quality Measures for Prostate Cancer Radiotherapy in the United States: Results from the Quality Research in Radiation Oncology (QRRO) Survey. , 2013, Practical Radiation Oncology.

[35]  Rajiv Bhatia,et al.  Growing the field of health impact assessment in the United States: an agenda for research and practice. , 2006, American journal of public health.

[36]  David A Hanauer,et al.  Leveraging EHR Data for Outcomes and Comparative Effectiveness Research in Oncology , 2012, Current Oncology Reports.

[37]  Jeremy M Grimshaw,et al.  A systematic review of the use of theory in the design of guideline dissemination and implementation strategies and interpretation of the results of rigorous evaluations , 2010, Implementation science : IS.

[38]  L. Sechrest,et al.  Approaches used in conducting health outcomes and effectiveness research , 2001 .

[39]  Franciscus A. van Vught,et al.  U-Multirank: design and testing the feasability of a multidimensional global university ranking: final report , 2011 .

[40]  G. Comi,et al.  Progressive MS Alliance Industry Forum: Maximizing Collective Impact To Enable Drug Development. , 2016, Trends in pharmacological sciences.

[41]  H. Davies,et al.  Using Evidence: How research can inform public services , 2007 .

[42]  J. Farrington,et al.  Ex ante impact assessment of policies affecting land use, Part B: application of the analytical framework , 2011 .

[43]  Dorothy H. Anderson,et al.  Using Stakeholders' Values to Apply Ecosystem Management in an Upper Midwest Landscape , 1999, Environmental management.

[44]  Fernando Antonio Basile Colugnati,et al.  A propensity score approach in the impact evaluation on scientific production in Brazilian biodiversity research: the BIOTA Program , 2014, Scientometrics.

[45]  E. Hazelkorn Assessing Europe's University-Based Research , 2010 .

[46]  J. Paty,et al.  Practical Considerations for the Use of Clinical Outcome Assessments (COAs) in Pediatric Clinical Research: Examples From Pediatric Gastroenterology , 2016, Therapeutic innovation & regulatory science.

[47]  B. V. D. Meulen Evaluating the societal relevance of academic research: A guide , 2010 .

[48]  M. Buxton,et al.  Getting Reearch into Practice: Assessing Benefits from Department of Health and National Health Service Research & Development , 2000 .

[49]  A. Hoeft,et al.  Standards for definitions and use of outcome measures for clinical effectiveness research in perioperative medicine: European Perioperative Clinical Outcome (EPCO) definitions A statement from the ESA-ESICM joint taskforce on perioperative outcome measures , 2015, European journal of anaesthesiology.

[50]  C Hodges Value for money? , 1990, Nursing times.

[51]  Gianluca Castelnuovo,et al.  A More Comprehensive Index in the Evaluation of Scientific Research: The Single Researcher Impact Factor Proposal , 2010, Clinical practice and epidemiology in mental health : CP & EMH.

[52]  Barry Bozeman,et al.  Public Value Mapping and Science Policy Evaluation , 2011 .

[53]  D. P. O'connor,et al.  Challenges in Outcome Measurement: Clinical Research Perspective , 2013, Clinical orthopaedics and related research.

[54]  S. Schaltegger A Framework for Ecopreneurship: Leading Bioneers and Environmental Managers to Ecopreneurship , 2002 .

[55]  Les Rymer,et al.  Measuring the Impact of Research--The Context for Metric Development. Go8 Backgrounder 23. , 2011 .

[56]  R. Freeman Strategic Management: A Stakeholder Approach , 2010 .

[57]  D. Tranfield,et al.  Towards a Methodology for Developing Evidence-Informed Management Knowledge by Means of Systematic Review , 2003 .

[58]  N. Haigh,et al.  Health impact assessment research and practice: A place for paradigm positioning? , 2012 .

[59]  Ethical Issues in Patient-Centered Outcomes Research and Comparative Effectiveness Research , 2015, Journal of empirical research on human research ethics : JERHRE.

[60]  The Canada Foundation for Innovation's outcome measurement study: a pioneering approach to research evaluation , 2010 .

[61]  Balaji R. Koka,et al.  The Reification of Absorptive Capacity: A Critical Review and Rejuvenation of the Construct , 2006 .

[62]  Stefan Seuring,et al.  From a literature review to a conceptual framework for sustainable supply chain management , 2008 .

[63]  Paul Jordan,et al.  Automated Research Impact Assessment: a new bibliometrics approach , 2015, Scientometrics.

[64]  Mywish K. Marediaa,et al.  Efficiency of research investments in the presence of international spillovers : wheat research in developing countries , 1999 .

[65]  Frank Vanclay,et al.  International Principles For Social Impact Assessment , 2003 .

[66]  Andrew J Milat,et al.  A narrative review of research impact assessment models and methods , 2015, Health Research Policy and Systems.

[67]  R. J. Hayes,et al.  Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. , 1995, JAMA.

[68]  Finn Hansson,et al.  Measuring research performance during a changing relationship between science and society , 2011 .

[69]  J. Molas-Gallart,et al.  Tracing productive interactions to identify social impacts: an example from the social sciences , 2011 .

[70]  Shyanika W. Rose,et al.  Toward the Assessment of Scientific and Public Health Impacts of the National Institute of Environmental Health Sciences Extramural Asthma Research Program Using Available Data , 2009, Environmental health perspectives.

[71]  Pirjo Kutinlahti,et al.  Research with an impact: Evaluation practices in public research organisations , 2006 .

[72]  Manfred Maier,et al.  Development of a practical tool to measure the impact of publications on the society based on focus group discussions with scientists , 2011, BMC public health.

[73]  Jonathan Grant,et al.  Measuring research: A guide to research evaluation frameworks and tools , 2013 .

[74]  D. Stryer,et al.  The outcomes of outcomes and effectiveness research: impacts and lessons from the first decade. , 2000, Health services research.

[75]  Alessandro Liberati,et al.  Conceptual frameworks and empirical approaches used to assess the impact of health research: an overview of reviews , 2011, Health research policy and systems.

[76]  J. Cousins,et al.  Assessing the Practice Impact of Research on Evaluation , 2015 .

[77]  Gillian E Cohen,et al.  Does health intervention research have real world policy and practice impacts: testing a new impact assessment tool , 2015, Health Research Policy and Systems.

[78]  Lutz Bornmann,et al.  Measuring the societal impact of research , 2012, EMBO reports.

[79]  Terttu Luukkonen,et al.  The difficulties in assessing the impact of EU framework programmes , 1998 .

[80]  C. Henshall The impact of Payback research: developing and using evidence in policy , 2011 .

[81]  R. McEachan,et al.  Action to Support Practices Implement Research Evidence (ASPIRE): protocol for a cluster-randomised evaluation of adaptable implementation packages targeting ‘high impact’ clinical practice recommendations in general practice , 2015, Implementation Science.

[82]  A. Jette,et al.  Uses of evidence in disability outcomes and effectiveness research. , 2002, The Milbank quarterly.

[83]  Dirk J. Roux,et al.  From scorecard to social learning: a reflective coassessment approach for promoting multiagency cooperation in natural resource management , 2011 .

[84]  M. Battaglia,et al.  Nonprofit foundations spur translational research. , 2014, Trends in pharmacological sciences.

[85]  A. Bauman,et al.  Policy and practice impacts of applied research: a case study analysis of the New South Wales Health Promotion Demonstration Research Grants Scheme 2000–2006 , 2013, Health Research Policy and Systems.

[86]  D. Czarnitzki,et al.  Value for Money? New Microeconometric Evidence on Public R&D Grants in Flanders , 2012 .

[87]  S. L. Manring,et al.  Strategy development in small and medium sized enterprises for sustainability and increased value creation , 2009 .

[88]  Stephen Hanney,et al.  Project Retrosight: Understanding the Returns from Cardiovascular and Stroke Research: The Policy Report. , 2011, Rand health quarterly.

[89]  J. Bridges,et al.  What outcomes should US policy makers compare in comparative effectiveness research? , 2010, Expert review of pharmacoeconomics & outcomes research.

[90]  Mark Harniss,et al.  The development of a clinical outcomes survey research application: Assessment CenterSM , 2010, Quality of Life Research.

[91]  Katharine Barker,et al.  The UK Research Assessment Exercise: the evolution of a national research evaluation system , 2007 .

[92]  Frederic S. Lee The Research Assessment Exercise, the state and the dominance of mainstream economics in British universities , 2006 .

[93]  Zhiqiang Wu,et al.  Average evaluation intensity: A quality-oriented indicator for the evaluation of research performance ☆ , 2015 .

[94]  Frank Vanclay,et al.  SOCIAL IMPACT ASSESSMENT: A CONTRIBUTION TO THE STATE OF THE ART SERIES , 1996 .

[95]  A. V. van Raan,et al.  Fatal attraction: Conceptual and methodological problems in the ranking of universities by bibliometric methods , 2005 .

[96]  A. Fink Conducting research literature reviews : from paper to the Internet , 1998 .

[97]  Henk F. Moed,et al.  The effect of "open access" on citation impact: An analysis of ArXiv's condensed matter section , 2006, J. Assoc. Inf. Sci. Technol..

[98]  Liz Allen,et al.  Tracking the impact of research on policy and practice: investigating the feasibility of using citations in clinical guidelines for research evaluation , 2012, BMJ Open.

[99]  C. Donovan,et al.  State of the art in assessing research impact: introduction to a special issue , 2011 .

[100]  C. Donovan,et al.  The Australian Research Quality Framework: A live experiment in capturing the social, economic, environmental, and cultural returns of publicly funded research , 2008 .

[101]  Arie Rip,et al.  Evaluation of societal quality of public sector research in the Netherlands , 2000 .

[102]  Ben R. Martin,et al.  Assessing Basic Research: The Case of the Isaac Newton Telescope , 1983 .

[103]  C. Weisel,et al.  Lessons learned from the application of BEES-C: Systematic assessment of study quality of epidemiologic research on BPA, neurodevelopment, and respiratory health. , 2015, Environment international.

[104]  Ben R. Martin,et al.  The Research Excellence Framework and the ‘impact agenda’: are we creating a Frankenstein monster? , 2011 .

[105]  Khim Horton,et al.  The Value of Nursing: a Literature Review , 2007, Nursing ethics.

[106]  Stefan Seuring,et al.  Conducting a Literature Review — The Example of Sustainability in Supply Chains , 2005 .

[107]  W. Riley,et al.  Enhancing measurement in health outcomes research supported by Agencies within the US Department of Health and Human Services , 2007, Quality of Life Research.

[108]  Antonio J. Gómez-Núñez,et al.  Impact oriented monitoring: A new methodology for monitoring and evaluation of international public health research projects , 2015 .

[109]  Alistair R. Anderson Cultivating the Garden of Eden: environmental entrepreneuring , 1998 .

[110]  S. Schaltegger A Framework for Ecopreneurship , 2002 .

[111]  D. Shepard,et al.  The Reinforcing Therapist Performance (RTP) experiment: Study protocol for a cluster randomized trial , 2010, Implementation science : IS.

[112]  Narongrit Sombatsompop,et al.  An evaluation of research performance for different subject categories using Impact Factor Point Average (IFPA) index: Thailand case study , 2005, Scientometrics.

[113]  Edward Nason,et al.  Health research: measuring the social, health and economic benefits , 2009, Canadian Medical Association Journal.

[114]  J. Britt Holbrook,et al.  Assessing the science-society relation: The case of the US National Science Foundation's second merit review criterion , 2005 .

[115]  J. B. Holbrook Re-assessing the science - society relation: The case of the US National Science Foundation's broader impacts merit review criterion (1997 - 2011) , 2012 .

[116]  David Roessner,et al.  Quantitative and qualitative methods and measures in the evaluation of research , 2000 .

[117]  P. V. D. Besselaar,et al.  Evaluation of research in context - an approach and two cases , 2011 .

[118]  J. Spaapen,et al.  Introducing ‘productive interactions’ in social impact assessment , 2011 .

[119]  Nabil Amara,et al.  New Evidence on Instrumental, Conceptual, and Symbolic Utilization of University Research in Government Agencies , 2004 .

[120]  C. Donovan The qualitative future of research evaluation , 2007 .

[121]  D. Pontille,et al.  The controversial policies of journal ratings: evaluating social sciences and humanities , 2010 .

[122]  Annette Evans,et al.  Assessment of sustainability indicators for renewable energy technologies , 2009 .

[123]  Anne-Wil Harzing,et al.  The Publish or Perish Book: Your guide to effective and responsible citation analysis , 2010 .

[124]  Kristi L Holmes,et al.  Beyond citation analysis: a model for assessment of research impact. , 2010, Journal of the Medical Library Association : JMLA.

[125]  P. Mayring Qualitative Content Analysis , 2000 .

[126]  Henk F. Moed,et al.  Multidimensional assessment of scholarly research impact , 2014, J. Assoc. Inf. Sci. Technol..

[127]  Fiorenzo Franceschini,et al.  Research quality evaluation: comparing citation counts considering bibliometric database errors , 2015 .

[128]  Margaret Anderson,et al.  On the path to a science of patient input , 2016, Science Translational Medicine.

[129]  R. Barré,et al.  S&T Indicators for Policy Making in a Changing Science-Society Relationship , 2004 .

[130]  J. Reichertz,et al.  Abduction: The Logic of Discovery of Grounded Theory , 2009 .

[131]  J. Schneider,et al.  Developing a methodology to assess the impact of research grant funding: a mixed methods approach. , 2014, Evaluation and program planning.

[132]  F. Leistritz,et al.  SOCIAL IMPACT ASSESSMENT , 1984 .

[133]  J. Benichou,et al.  Reading factor: a new bibliometric criterion for managing digital libraries. , 2002, Journal of the Medical Library Association : JMLA.

[134]  Lutz Bornmann,et al.  How should the societal impact of research be generated and measured? A proposal for a simple and practicable approach to allow interdisciplinary comparisons , 2013, Scientometrics.

[135]  A. Boyd,et al.  Frameworks for evaluating health research capacity strengthening: a qualitative study , 2013, Health Research Policy and Systems.

[136]  J. Britt Holbrook,et al.  Comparative Assessment of Peer Review (CAPR): EU/US workshop on peer review: Assessing "broader impact" in research grant applications , 2010 .

[137]  Nick Black,et al.  The Cooksey review of UK health research funding , 2006, BMJ : British Medical Journal.

[138]  O. Ciccarelli,et al.  Setting a research agenda for progressive multiple sclerosis: The International Collaborative on Progressive MS , 2012, Multiple sclerosis.

[139]  Anastassios Pouris,et al.  Scientometric impact assessment of a research policy instrument: the case of rating researchers on scientific outputs in South Africa , 2011, Scientometrics.

[140]  Ammon Salter,et al.  Measuring third stream activities , 2002 .

[141]  Massimo Stafoggia,et al.  P.Re.Val.E.: outcome research program for the evaluation of health care quality in Lazio, Italy , 2012, BMC Health Services Research.

[142]  David Fryer,et al.  Review: Erica Burman (2008). Deconstructing developmental psychology , 2009 .

[143]  J. Brewer,et al.  The impact of impact , 2011 .

[144]  Joost Dekker,et al.  DOING THE RIGHT THING AND DOING IT RIGHT: TOWARD A FRAMEWORK FOR ASSESSING THE POLICY RELEVANCE OF HEALTH SERVICES RESEARCH , 2003, International Journal of Technology Assessment in Health Care.

[145]  Alicia C. Bunger,et al.  Outcomes for Implementation Research: Conceptual Distinctions, Measurement Challenges, and Research Agenda , 2010, Administration and Policy in Mental Health and Mental Health Services Research.