Considerations of complexity in rating certainty of evidence in systematic reviews: a primer on using the GRADE approach in global health

Public health interventions and health technologies are commonly described as ‘complex’, as they involve multiple interacting components and outcomes, and their effects are largely influenced by contextual interactions and system-level processes. Systematic reviewers and guideline developers evaluating the effects of these complex interventions and technologies report difficulties in using existing methods and frameworks, such as the Grading of Recommendations Assessment, Development and Evaluation (GRADE). As part of a special series of papers on implications of complexity in the WHO guideline development, this paper serves as a primer on how to consider sources of complexity when using the GRADE approach to rate certainty of evidence. Relevant sources of complexity in systematic reviews, health technology assessments and guidelines of public health are outlined and mapped onto the reported difficulties in rating the estimates of the effect of these interventions. Recommendations on how to address these difficulties are further outlined, and the need for an integrated use of GRADE from the beginning of the review or guideline development is emphasised. The content of this paper is informed by the existing GRADE guidance, an ongoing research project on considering sources of complexity when applying the GRADE approach to rate certainty of evidence in systematic reviews and the review authors’ own experiences with using GRADE.

[1]  D. Peters,et al.  Better Guidance Is Welcome, but without Blinders , 2012, PLoS medicine.

[2]  S. Grant,et al.  Reviewing and interpreting the effects of brief alcohol interventions: comment on a Cochrane review about motivational interviewing for young adults , 2015, Addiction.

[3]  James Thomas,et al.  Developing and Optimising the Use of Logic Models in Systematic Reviews: Exploring Practice and Good Practice in the Use of Programme Theory in Reviews , 2015, PloS one.

[4]  Meera Viswanathan,et al.  AHRQ series on complex intervention systematic reviews-paper 6: PRISMA-CI extension statement and checklist. , 2017, Journal of clinical epidemiology.

[5]  P Garner,et al.  Deworming drugs for treating soil-transmitted intestinal worms in children: effects on growth and school performance. , 2007, The Cochrane database of systematic reviews.

[6]  G. Melendez‐Torres,et al.  Outcomes in systematic reviews of complex interventions never reached "high" GRADE ratings when compared with those of simple interventions. , 2016, Journal of clinical epidemiology.

[7]  D. Moher,et al.  Guidance for Developers of Health Research Reporting Guidelines , 2010, PLoS medicine.

[8]  Louise Brereton,et al.  Guidance on the integrated assessment of complex health technologies - The INTEGRATE-HTA Model , 2016 .

[9]  P. Tugwell,et al.  Guidance for Evidence-Informed Policies about Health Systems: Assessing How Much Confidence to Place in the Research Evidence , 2012, PLoS medicine.

[10]  N. Sewankambo,et al.  Towards a common definition of global health , 2009, The Lancet.

[11]  R. Thomson,et al.  Decision aids for people facing health treatment or screening decisions. , 2003, The Cochrane database of systematic reviews.

[12]  P. Ubel,et al.  Barriers to influenza immunization in a low-income urban population. , 2001, American journal of preventive medicine.

[13]  Rachel Churchill,et al.  Introducing a series of methodological articles on considering complexity in systematic reviews of interventions. , 2013, Journal of clinical epidemiology.

[14]  Susan Michie,et al.  Developing and Evaluating Complex Interventions , 2015 .

[15]  Evelyn P. Whitlock,et al.  Screening and Interventions for Childhood Overweight: A Summary of Evidence for the US Preventive Services Task Force , 2005, Pediatrics.

[16]  Deborah M Caldwell,et al.  Synthesising quantitative evidence in systematic reviews of complex health interventions , 2019, BMJ Global Health.

[17]  E. Evangelou,et al.  Disclosure of researcher allegiance in meta-analyses and randomised controlled trials of psychotherapy: a systematic appraisal , 2015, BMJ Open.

[18]  P. Tugwell,et al.  Mass deworming to improve developmental health and wellbeing of children in low-income and middle-income countries: a systematic review and network meta-analysis. , 2017, The Lancet. Global health.

[19]  Diana Petitti,et al.  Update on the Methods of the U.S. Preventive Services Task Force: Estimating Certainty and Magnitude of Net Benefit , 2007, Annals of Internal Medicine.

[20]  Kate Flemming,et al.  Taking account of context in systematic reviews and guidelines considering a complexity perspective , 2019, BMJ Global Health.

[21]  France Légaré,et al.  Decision aids for people facing health treatment or screening decisions. , 2017, The Cochrane database of systematic reviews.

[22]  Alain Mayhew,et al.  Decisions about lumping vs. splitting of the scope of systematic reviews of complex interventions are not well justified: a case study in systematic reviews of health care professional reminders. , 2012, Journal of clinical epidemiology.

[23]  E. Rehfuess,et al.  Rating the quality of a body of evidence on the effectiveness of health and social interventions: A systematic review and mapping of evidence domains , 2018, Research synthesis methods.

[24]  Randy W. Elder,et al.  Complex interventions and their implications for systematic reviews: a pragmatic approach. , 2013, Journal of clinical epidemiology.

[25]  N. Ford,et al.  Strength of recommendations in WHO guidelines using GRADE was associated with uptake in national policy. , 2015, Journal of clinical epidemiology.

[26]  R. Armstrong,et al.  TIDieR-PHP: a reporting guideline for population health and policy interventions , 2018, British Medical Journal.

[27]  G. Guyatt,et al.  GRADE guidelines: 18. How ROBINS-I and other tools to assess risk of bias in nonrandomized studies should be used to rate the certainty of a body of evidence. , 2018, Journal of clinical epidemiology.

[28]  Eva A Rehfuess,et al.  Quality appraisal in systematic reviews of public health interventions: an empirical study on the impact of choice of tool on meta-analysis , 2012, Journal of Epidemiology & Community Health.

[29]  Alan Shiell,et al.  Theorising Interventions as Events in Systems , 2009, American journal of community psychology.

[30]  Janet E Squires,et al.  Systematic reviews of complex interventions: framing the review question. , 2013, Journal of clinical epidemiology.

[31]  Andrew Booth,et al.  Towards a taxonomy of logic models in systematic reviews and health technology assessments: A priori, staged, and iterative approaches , 2018, Research synthesis methods.

[32]  Howard Balshem,et al.  GRADE guidelines: 3. Rating the quality of evidence. , 2011, Journal of clinical epidemiology.

[33]  Meera Viswanathan,et al.  Agency for Healthcare Research and Quality Evidence-based Practice Center methods for systematically reviewing complex multicomponent health care interventions. , 2014, Journal of clinical epidemiology.

[34]  Gordon H Guyatt,et al.  GRADE guidelines: 9. Rating up the quality of evidence. , 2011, Journal of clinical epidemiology.

[35]  Elie A Akl,et al.  Current experience with applying the GRADE approach to public health interventions: an empirical study , 2013, BMC Public Health.

[36]  M. Hernán,et al.  ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions , 2016, British Medical Journal.

[37]  Andrew Booth,et al.  Applying GRADE-CERQual to qualitative evidence synthesis findings: introduction to the series , 2018, Implementation Science.

[38]  Jacob Burns,et al.  Environmental interventions to reduce the consumption of sugar‐sweetened beverages and their effects on health , 2016, The Cochrane database of systematic reviews.

[39]  Deborah M Caldwell,et al.  Approaches for synthesising complex mental health interventions in meta-analysis , 2016, Evidence-Based Mental Health.

[40]  Mary Lyn Gaffield,et al.  Grading of Recommendations Assessment, Development, and Evaluation (GRADE) , 2011 .

[41]  Andrew Booth,et al.  Making sense of complexity in context and implementation: the Context and Implementation of Complex Interventions (CICI) framework , 2017, Implementation Science.

[42]  Gordon H Guyatt,et al.  GRADE guidelines: 7. Rating the quality of evidence--inconsistency. , 2011, Journal of clinical epidemiology.

[43]  David Ogilvie,et al.  Systematic reviews of health effects of social interventions: 2. Best available evidence: how low should you go? , 2005, Journal of Epidemiology and Community Health.

[44]  Robert Fletcher,et al.  Deworming and adjuvant interventions for improving the developmental health and well-being of children in low-and middle-income countries : a systematic review and network meta-analysis , 2015 .

[45]  G. Melendez‐Torres,et al.  Users identified challenges in applying GRADE to complex interventions and suggested an extension to GRADE. , 2016, Journal of clinical epidemiology.

[46]  C. May.  Towards a general theory of implementation , 2013, Implementation Science.

[47]  Jonathan J Deeks,et al.  Issues relating to study design and risk of bias when including non‐randomized studies in systematic reviews on the effects of interventions , 2013, Research synthesis methods.

[48]  Mark Petticrew,et al.  Time to rethink the systematic review catechism? Moving from ‘what works’ to ‘what happens’ , 2015, Systematic Reviews.

[49]  S. Grant,et al.  It is time to develop appropriate tools for assessing minimal clinically important differences, performance bias and quality of evidence in reviews of behavioral interventions. , 2016, Addiction.

[50]  J. Wyatt,et al.  Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide , 2014, BMJ : British Medical Journal.

[51]  G. Guyatt,et al.  GRADE guidelines: 4. Rating the quality of evidence--study limitations (risk of bias). , 2011, Journal of clinical epidemiology.

[52]  D. Foxcroft.  We cannot ignore bias, especially if effects are small, but we need better methods for evaluating prevention systems. , 2016, Addiction.

[53]  G. Guyatt,et al.  GRADE: an emerging consensus on rating quality of evidence and strength of recommendations , 2008, BMJ : British Medical Journal.

[54]  Mohammed T Ansari,et al.  The GRADE Working Group clarifies the construct of certainty of evidence. , 2017, Journal of clinical epidemiology.

[55]  H. Schünemann.  Methodological idiosyncracies, frameworks and challenges of non-pharmaceutical and non-technical treatment interventions. , 2013, Zeitschrift fur Evidenz, Fortbildung und Qualitat im Gesundheitswesen.

[56]  H. Folgering,et al.  Pulmonary rehabilitation in chronic obstructive pulmonary disease. , 1998, The European respiratory journal.

[57]  Eva Rehfuess,et al.  Slum upgrading strategies involving physical environment and infrastructure interventions and their effects on health and socio-economic outcomes. , 2013, The Cochrane database of systematic reviews.

[58]  R. Gold,et al.  Screening and Interventions for Childhood Overweight , 2005 .

[59]  C. Mulrow,et al.  Current methods of the US Preventive Services Task Force: a review of the process. , 2001, American journal of preventive medicine.

[60]  D. Pieper,et al.  Heterogeneity in application, design, and analysis characteristics was found for controlled before-after and interrupted time series studies included in Cochrane reviews. , 2017, Journal of clinical epidemiology.

[61]  M. Petticrew,et al.  Developing and evaluating complex interventions: the new Medical Research Council guidance , 2008, BMJ : British Medical Journal.

[62]  D. Yach,et al.  Global priority setting for Cochrane systematic reviews of health promotion and public health research , 2005, Journal of Epidemiology and Community Health.

[63]  Jane Noyes,et al.  Implications of a complexity perspective for systematic reviews and guideline development in health decision making , 2019, BMJ Global Health.

[64]  Gordon H Guyatt,et al.  GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 1: Introduction , 2016, British Medical Journal.

[65]  Justin Parkhurst,et al.  What Constitutes “Good” Evidence for Public Health and Social Policy-making? From Hierarchies to Appropriateness , 2016 .

[66]  G. Guyatt,et al.  GRADE guidelines: 8. Rating the quality of evidence--indirectness. , 2011, Journal of clinical epidemiology.

[67]  Susan L Norris,et al.  The WHO-INTEGRATE evidence to decision framework version 1.0: integrating WHO norms and values and a complexity perspective , 2019, BMJ Global Health.

[68]  Andrew Booth,et al.  Series: Clinical Epidemiology in South Africa. Paper 3: Logic models help make sense of complexity in systematic reviews and health technology assessments. , 2017, Journal of clinical epidemiology.

[69]  Susan Michie,et al.  Assessing the complexity of interventions within systematic reviews: development, content and use of a new tool (iCAT_SR) , 2017, BMC Medical Research Methodology.

[70]  I. Bogoch,et al.  The benefits of mass deworming on health outcomes: new evidence synthesis, the debate persists. , 2017, The Lancet. Global health.

[71]  H. Schünemann,et al.  Rating the certainty in evidence in the absence of a single estimate of effect , 2017, Evidence-Based Medicine.