The WHO-INTEGRATE evidence to decision framework version 1.0: integrating WHO norms and values and a complexity perspective

Introduction Evidence-to-decision (EtD) frameworks intend to ensure that all criteria of relevance to a health decision are systematically considered. This paper, part of a series commissioned by the WHO, reports on the development of an EtD framework that is rooted in WHO norms and values, reflective of the changing global health landscape, and suitable for a range of interventions and complexity features. We also sought to assess the value of this framework to decision-makers at global and national levels, and to facilitate uptake through suggestions on how to prioritise criteria and methods to collect evidence. Methods In an iterative, principles-based approach, we developed the framework structure from WHO norms and values. Preliminary criteria were derived from key documents and supplemented with comprehensive subcriteria obtained through an overview of systematic reviews of criteria employed in health decision-making. We assessed to what extent the framework can accommodate features of complexity, and conducted key informant interviews among WHO guideline developers. Suggestions on methods were drawn from the literature and expert consultation. Results The new WHO-INTEGRATE (INTEGRATe Evidence) framework comprises six substantive criteria—balance of health benefits and harms, human rights and sociocultural acceptability, health equity, equality and non-discrimination, societal implications, financial and economic considerations, and feasibility and health system considerations—and the meta-criterion quality of evidence. It is intended to facilitate a structured process of reflection and discussion in a problem-specific and context-specific manner from the start of a guideline development or other health decision-making process. For each criterion, the framework offers a definition, subcriteria and example questions; it also suggests relevant primary research and evidence synthesis methods and approaches to assessing quality of evidence. Conclusion The framework is deliberately labelled version 1.0. We expect further modifications based on focus group discussions in four countries, example applications and input across concerned disciplines.

[1]  Becky Skidmore,et al.  Priority setting for health technology assessments: A systematic review of current practical approaches , 2007, International Journal of Technology Assessment in Health Care.

[2]  N. Daniels,et al.  Limits to health care: fair procedures, democratic deliberation, and the legitimacy problem for insurers. , 1997, Philosophy & public affairs.

[3]  Zafar Mirza,et al.  Strengthening health systems for universal health coverage and sustainable development , 2017, Bulletin of the World Health Organization.

[4]  Luke Vale,et al.  Issues in the incorporation of economic perspectives and evidence into Cochrane reviews , 2013, Systematic Reviews.

[5]  Bjørn Hofmann,et al.  HARMONIZATION OF ETHICS IN HEALTH TECHNOLOGY ASSESSMENT: A REVISION OF THE SOCRATIC APPROACH , 2014, International Journal of Technology Assessment in Health Care.

[6]  Andrew Booth,et al.  Applying GRADE-CERQual to qualitative evidence synthesis findings: introduction to the series , 2018, Implementation Science.

[7]  Alexandra Robbins,et al.  The World Health Report 2000: Health Systems: Improving Performance. Geneva: World Health Organization; 2000. 150 pages plus tables; $13.50; available on-line from: URL: http://www.who.int/whr/ , 2001 .

[8]  G. Guyatt,et al.  [GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 2: Clinical practice guidelines]. , 2018, Gaceta sanitaria.

[9]  S. Evers,et al.  How to prepare a systematic review of economic evaluations for informing evidence-based healthcare decisions: data extraction, risk of bias, and transferability (part 3/3) , 2016, Expert review of pharmacoeconomics & outcomes research.

[10]  Elie A Akl,et al.  Current experience with applying the GRADE approach to public health interventions: an empirical study , 2013, BMC Public Health.

[11]  Elena Savoia,et al.  Integrating emergency risk communication (ERC) into the public health system response: Systematic review of literature to aid formulation of the 2017 WHO Guideline for ERC policy and practice , 2018, PloS one.

[12]  G. Guyatt,et al.  GRADE: an emerging consensus on rating quality of evidence and strength of recommendations , 2008, BMJ : British Medical Journal.

[13]  S. Tantivess,et al.  Coverage Decisions and the Court: A Public Health Perspective on Glucosamine Reimbursement in Thailand , 2016, Health systems and reform.

[14]  Gerald Gartlehner,et al.  [GRADE guidelines: 10. Considering resource use and rating the quality of economic evidence]. , 2013, Zeitschrift fur Evidenz, Fortbildung und Qualitat im Gesundheitswesen.

[15]  Gerald Gartlehner,et al.  [Communicating Risk in Public Health Emergencies: A WHO Guideline for Emergency Risk Communication (Erc) Policy and Practice]. , 2019, Gesundheitswesen (Bundesverband der Arzte des Offentlichen Gesundheitsdienstes (Germany)).

[16]  M. Becker Evidence Argument And Persuasion In The Policy Process , 2016 .

[17]  Ajay Mahal,et al.  Stated and Revealed Preferences for Funding New High-Cost Cancer Drugs: A Critical Review of the Evidence from Patients, the Public and Payers , 2016, The Patient - Patient-Centered Outcomes Research.

[18]  Mickaël Hiligsmann,et al.  How to prepare a systematic review of economic evaluations for informing evidence-based healthcare decisions: a five-step approach (part 1/3) , 2016, Expert review of pharmacoeconomics & outcomes research.

[19]  Johan P Mackenbach,et al.  An overview of ethical frameworks in public health: can they be supportive in the evaluation of programs to prevent overweight? , 2010, BMC public health.

[20]  Abdullah Pandor,et al.  The use of rapid review methods in health technology assessments: 3 case studies , 2016, BMC Medical Research Methodology.

[21]  Sarah E. Rosenbaum,et al.  Policymakers’ and other stakeholders’ perceptions of key considerations for health system decisions and the presentation of evidence to inform those considerations: an international survey , 2013, Health Research Policy and Systems.

[22]  Hanane Khoury,et al.  Bridging Health Technology Assessment (HTA) and Efficient Health Care Decision Making with Multicriteria Decision Analysis (MCDA) , 2012, Medical decision making : an international journal of the Society for Medical Decision Making.

[23]  P. Byass Systems thinking for health systems strengthening , 2011 .

[24]  A. Rothberg Making Choices in Health: WHO Guide to Cost Effectiveness Analysis , 2008 .

[25]  Jane Noyes,et al.  Implications of a complexity perspective for systematic reviews and guideline development in health decision making , 2019, BMJ Global Health.

[26]  Peter Hall,et al.  The right to the highest attainable standard of health , 2015, The Lancet.

[27]  Guy M. Goodwin,et al.  Introduction to Systematic Reviews , 2004, Journal of psychopharmacology.

[28]  Kate Flemming,et al.  Qualitative evidence synthesis for complex interventions and guideline development: clarification of the purpose, designs and relevant methods , 2019, BMJ Global Health.

[29]  Marc Fleurbaey,et al.  Guidance on priority setting in health care (GPS-Health): the inclusion of equity criteria not captured by cost-effectiveness analysis , 2014, Cost Effectiveness and Resource Allocation.

[30]  Katherine E. Smith,et al.  A glossary of theories for understanding policymaking , 2012, Journal of Epidemiology & Community Health.

[31]  Vivian A. Welcha,et al.  GRADE equity guidelines 3 : health equity considerations in rating the certainty of synthesized evidence , 2017 .

[32]  Geoffrey R Swain,et al.  Preparedness: medical ethics versus public health ethics. , 2008, Journal of public health management and practice : JPHMP.

[33]  Kate Flemming,et al.  Synthesising quantitative and qualitative evidence to inform guidelines on complex interventions: clarifying the purposes, designs and outlining some methods , 2019, BMJ Global Health.

[34]  S. Athar Principles of Biomedical Ethics , 2011, The Journal of IMA.

[35]  Sofia Tranæus,et al.  FRAMEWORK FOR SYSTEMATIC IDENTIFICATION OF ETHICAL ASPECTS OF HEALTHCARE TECHNOLOGIES: THE SBU APPROACH , 2015, International Journal of Technology Assessment in Health Care.

[36]  Lisa Hartling,et al.  What guidance is available for researchers conducting overviews of reviews of healthcare interventions? A scoping review and qualitative metasummary , 2016, Systematic Reviews.

[37]  Andrew Booth,et al.  Making sense of complexity in context and implementation: the Context and Implementation of Complex Interventions (CICI) framework , 2017, Implementation Science.

[38]  Tim Evans,et al.  Applying an equity lens to interventions: using PROGRESS ensures consideration of socially stratifying factors to illuminate inequities in health. , 2014, Journal of clinical epidemiology.

[39]  J. Higgins,et al.  Cochrane Handbook for Systematic Reviews of Interventions , 2010, International Coaching Psychology Review.

[40]  Iñaki Gutiérrez-Ibarluzea,et al.  Q-SEA – a tool for quality assessment of ethics analyses conducted as part of health technology assessments , 2017, GMS health technology assessment.

[41]  Yukiko Asada,et al.  The Promise of Public Health: Ethical Reflections , 2006, Canadian journal of public health = Revue canadienne de sante publique.

[42]  Paul Kind,et al.  From efficacy to equity: Literature review of decision criteria for resource allocation and healthcare decisionmaking , 2012, Cost Effectiveness and Resource Allocation.

[43]  N. Tromp,et al.  Progressive realisation of universal health coverage: what are the required processes and evidence? , 2017, BMJ Global Health.

[44]  Rob Baltussen,et al.  Mapping of multiple criteria for priority setting of health interventions: an aid for decision makers , 2012, BMC Health Services Research.

[45]  H. Arksey,et al.  Scoping studies: towards a methodological framework , 2005 .

[46]  Sarah E. Rosenbaum,et al.  Developing and evaluating communication strategies to support informed decisions and practice based on evidence (DECIDE): protocol and preliminary results , 2013, Implementation Science.

[47]  Ross Upshur,et al.  Principles for the Justification of Public Health Intervention , 2002, Canadian journal of public health = Revue canadienne de sante publique.

[48]  Mohammadreza Mobinizadeh,et al.  The health systems' priority setting criteria for selecting health technologies: A systematic review of the current evidence , 2016, Medical journal of the Islamic Republic of Iran.

[49]  H. Schünemann,et al.  [GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 1: Introduction.] , 2017, Recenti progressi in medicina.

[50]  Peter Schröder-Bäck,et al.  Ethics in public health: call for shared moral public health literacy. , 2017, European journal of public health.

[51]  Susan Michie,et al.  Assessing the complexity of interventions within systematic reviews: development, content and use of a new tool (iCAT_SR) , 2017, BMC Medical Research Methodology.

[52]  D. Colombo,et al.  Closing the gap in a generation: health equity through action on the social determinants of health. Final report of the Commission on Social Determinants of Health , 2010 .

[53]  Rob Baltussen,et al.  Priority Setting for Universal Health Coverage: We Need Evidence-Informed Deliberative Processes, Not Just More Evidence on Cost-Effectiveness , 2016, International journal of health policy and management.

[54]  Andrew Tannahill,et al.  Beyond evidence--to ethics: a decision-making framework for health promotion, public health and health improvement. , 2008, Health promotion international.

[55]  W. V. Lerberghe,et al.  The World Health Report 2008 Primary Health Care – Now More Than Ever 100 Adapting reforms to country context , 2008 .

[56]  A. Mills,et al.  National decision-making on adopting new vaccines: a systematic review. , 2012, Health policy and planning.

[57]  I. Sanderson,et al.  Intelligent Policy Making for a Complex World: Pragmatism, Evidence and Learning , 2009 .

[58]  Daniel Strech,et al.  Current state of ethics literature synthesis: a systematic review of reviews , 2016, BMC Medicine.

[59]  Karen Faith,et al.  Pandemic influenza preparedness: an ethical framework to guide decision-making , 2006, BMC medical ethics.

[60]  Denis Porignon,et al.  Policy dialogue: What it is and how it can contribute to evidence-informed decision-making. , 2015 .

[61]  Lisa M. Lee Public Health Ethics Theory: Review and Path to Convergence , 2012, Journal of Law, Medicine & Ethics.

[62]  Mark Nuijten,et al.  Principles of good practice for budget impact analysis: report of the ISPOR Task Force on good research practices--budget impact analysis. , 2007, Value in health : the journal of the International Society for Pharmacoeconomics and Outcomes Research.

[63]  Carlo Petrini,et al.  Theoretical Models and Operational Frameworks in Public Health Ethics , 2010, International journal of environmental research and public health.

[64]  L. Gostin,et al.  Public Health Ethics: Mapping the Terrain , 2002, The Journal of law, medicine & ethics : a journal of the American Society of Law, Medicine & Ethics.

[65]  Matthias Briel,et al.  GRADE equity guidelines 3: considering health equity in GRADE guideline development: rating the certainty of synthesized evidence , 2017, Journal of clinical epidemiology.

[66]  E. Akl,et al.  The GRADE evidence-to-decision framework: a report of its testing and application in 15 international guideline panels , 2015, Implementation Science.

[67]  Kate Flemming,et al.  Taking account of context in systematic reviews and guidelines considering a complexity perspective , 2019, BMJ Global Health.

[68]  Jennie Popay,et al.  Women’s and Children’s Health: Evidence of Impact of Human Rights , 2013 .

[69]  Mohammed T Ansari,et al.  The GRADE Working Group clarifies the construct of certainty of evidence. , 2017, Journal of clinical epidemiology.

[70]  Ishani Kar-Purkayastha,et al.  Public health: ethical issues. , 2009 .

[71]  Kate Flemming,et al.  Formulating questions to explore complex interventions within qualitative evidence synthesis , 2019, BMJ Global Health.

[72]  Matthias Briel,et al.  GRADE equity guidelines 4: considering health equity in GRADE guideline development: evidence to decision process. , 2017, Journal of clinical epidemiology.

[73]  Narasimhan Manjulaa,et al.  Consolidated guideline on sexual and reproductive health and rights of women living with HIV. , 2017 .

[74]  Charalabos-Markos Dintsios,et al.  Information on ethical issues in health technology assessment: How and where to find them , 2010, International Journal of Technology Assessment in Health Care.

[75]  Gordon H Guyatt,et al.  Decision-making frameworks and considerations for informing coverage decisions for healthcare interventions: a critical interpretive synthesis. , 2017, Journal of clinical epidemiology.

[76]  N. Kass An ethics framework for public health. , 2001, American journal of public health.

[77]  Paul Montgomery,et al.  Considerations of complexity in rating certainty of evidence in systematic reviews: a primer on using the GRADE approach in global health , 2019, BMJ Global Health.

[78]  Alireza Zali,et al.  The evolution of public health ethics frameworks: systematic review of moral values and norms in public health policy , 2017, Medicine, Health Care and Philosophy.

[79]  Peter D. Jacobson,et al.  Looking Ahead: Addressing Ethical Challenges in Public Health Practice , 2007, Journal of Law, Medicine & Ethics.

[80]  Lydia Kapiriri,et al.  Setting priorities for health interventions in developing countries: a review of empirical studies , 2009, Tropical medicine & international health : TM & IH.

[81]  P. Mayring Qualitative content analysis: theoretical foundation, basic procedures and software solution , 2014 .

[82]  S. Wilson Methods for the economic evaluation of health care programmes , 1987 .

[83]  Daniel Strech,et al.  Putting Public Health Ethics into Practice: A Systematic Framework , 2015, Front. Public Health.

[84]  David Moher,et al.  Evidence summaries: the evolution of a rapid review approach , 2012, Systematic Reviews.

[85]  L. Erickson,et al.  Evidence and Value: Impact on DEcisionMaking – the EVIDEM framework and potential applications , 2008, BMC health services research.

[86]  Alice Welbourn,et al.  Using GRADE as a framework to guide research on the sexual and reproductive health and rights (SRHR) of women living with HIV – methodological opportunities and challenges , 2017, AIDS care.

[87]  Aubrey Sheiham,et al.  Closing the gap in a generation: health equity through action on the social determinants of health. A report of the WHO Commission on Social Determinants of Health (CSDH) 2008. , 2009, Community dental health.

[88]  Andrew Booth,et al.  Using Qualitative Evidence in Decision Making for Health and Social Interventions: An Approach to Assess Confidence in Findings from Qualitative Evidence Syntheses (GRADE-CERQual) , 2015, PLoS medicine.

[89]  Rob Baltussen,et al.  Translating international HIV treatment guidelines into local priorities in Indonesia , 2018, Tropical medicine & international health : TM & IH.

[90]  Ole Frithjof Norheim,et al.  CAN COST-EFFECTIVENESS ANALYSIS INTEGRATE CONCERNS FOR EQUITY? SYSTEMATIC REVIEW , 2012, International Journal of Technology Assessment in Health Care.

[91]  Marcia Tummers,et al.  Value Assessment Frameworks for HTA Agencies: The Organization of Evidence-Informed Deliberative Processes. , 2017, Value in health : the journal of the International Society for Pharmacoeconomics and Outcomes Research.

[92]  J. Kleijnen,et al.  How to prepare a systematic review of economic evaluations for clinical practice guidelines: database selection and search strategy development (part 2/3) , 2016, Expert review of pharmacoeconomics & outcomes research.

[93]  C. Mitton,et al.  ‘Real-world’ health care priority setting using explicit decision criteria: a systematic review of the literature , 2015, BMC Health Services Research.