Systematic reviews: from ‘bare bones’ reviews to policy relevance

Theory-based systematic reviews, which summarise evidence on what works, when and why, strive for greater policy relevance. Reviews that answer these questions adopt a mixed methods approach and draw on a range of study types. Answering the ‘what works’ and ‘what doesn't’ questions means drawing on effectiveness studies, conducted to standards of high-quality impact evaluation. But in formulating answers to the ‘when’ and ‘why’ questions requires a broader range of evidence from both quantitative and qualitative research. Based on a review of the methodological literature in this field and the experience of 3ie's systematic reviews program, this article provides an outline of how a theory-based approach to systematic reviews, including appropriate quantitative and qualitative evidence, can be operationalised. We propose an approach based on three principles: (1) develop a program theory for the intervention; (2) adopt a mixed methods approach and include a broader range of evidence; and (3) maintain the rigour and transparency that characterise systematic reviews. The approach translates into two broad options. Effectiveness plus reviews focus on providing a detailed causal chain analysis by drawing on a program theory and additional data collection on context and intervention implementation. And effectiveness plus with a parallel review component, which is designed to answer specific research questions related to effectiveness, adopts separate inclusion criteria, reflecting the type of studies appropriate for answering those questions.

[1]  Jennie Popay,et al.  Qualitative Research and Cochrane Reviews , 2008 .

[2]  Carol Hirschon Weiss,et al.  Theory-Based Evaluation in Practice , 2000, Evaluation review.

[3]  T. Cook,et al.  Quasi-experimentation: Design & analysis issues for field settings , 1979 .

[4]  M. Kelly,et al.  Synthesizing diverse evidence: the use of primary qualitative data analysis methods and logic models in public health reviews. , 2010, Public health.

[5]  Sandy Oliver,et al.  Integrating qualitative research with trials in systematic reviews , 2004, BMJ : British Medical Journal.

[6]  Margarete Sandelowski,et al.  The Challenges of Searching for and Retrieving Qualitative Studies , 2003, Western journal of nursing research.

[7]  J. Glanville,et al.  Searching for Studies , 2008 .

[8]  Sandy Oliver,et al.  An Emerging Framework for Including Different Types of Evidence in Systematic Reviews for Public Policy , 2005 .

[9]  Evelyn P. Whitlock,et al.  Screening and Interventions for Childhood Overweight: A Summary of Evidence for the US Preventive Services Task Force , 2005, Pediatrics.

[10]  Carolyn Turpin-Petrosino,et al.  Scared Straight and Other Juvenile Awareness Programs for Preventing Juvenile Delinquency: A Systematic Review , 2013 .

[11]  D. Anthony.  Evidence-based Policy: A Realist Perspective , 2007 .

[12]  Rob W Anderson,et al.  The Role of Economic Perspectives and Evidence in Systematic Review , 2010 .

[13]  Ruth Stewart,et al.  What is the impact of microfinance on poor people?: a systematic review of evidence from sub-Saharan Africa , 2010 .

[14]  Michele Tarsilla.  Cochrane Handbook for Systematic Reviews of Interventions , 2010, Journal of MultiDisciplinary Evaluation.

[15]  Giel Ton,et al.  The mixing of methods: A three-step process for improving rigour in impact evaluations , 2012 .

[16]  M. Petticrew,et al.  Limits to modern contraceptive use among young women in developing countries: a systematic review of qualitative research , 2009, Reproductive health.

[17]  Thomas J. Cook,et al.  Avoiding type III error in program evaluation: Results from a field experiment , 1980 .

[18]  Janet Harris.  Supplementary Guidance for Inclusion of Qualitative Research in Cochrane Systematic Reviews of Interventions , 2011 .

[19]  V. Welch,et al.  Realist review to understand the efficacy of culturally appropriate diabetes education programmes , 2013, Diabetic medicine : a journal of the British Diabetic Association.

[20]  Angela Harden,et al.  Synthesis: combining results systematically and appropriately , 2012 .

[21]  R. Berg,et al.  Interventions to reduce the prevalence of female genital mutilation/cutting in African countries , 2011 .

[22]  Robert D Herbert,et al.  Analysis of quality of interventions in systematic reviews , 2005, BMJ : British Medical Journal.

[23]  Evan Mayo-Wilson,et al.  Reporting implementation in randomized trials: proposed additions to the consolidated standards of reporting trials statement. , 2007, American journal of public health.

[24]  L. Delbeke.  Quasi-experimentation - design and analysis issues for field settings - cook,td, campbell,dt , 1980 .

[25]  Tina Miller,et al.  Finding qualitative research: an evaluation of search strategies , 2004, BMC medical research methodology.

[26]  Sandy Oliver,et al.  Children and healthy eating: a systematic review of barriers and facilitators , 2003 .

[27]  Margarete Sandelowski,et al.  Making Sense of Qualitative and Quantitative Findings in Mixed Research Synthesis Studies , 2008, Field methods.

[28]  Adam Fletcher,et al.  Teenage pregnancy and social disadvantage: systematic review integrating controlled trials and qualitative studies , 2009, BMJ : British Medical Journal.

[29]  J. Popay,et al.  Systematically reviewing qualitative and quantitative evidence to inform management and policy-making in the health field , 2005, Journal of health services research & policy.

[30]  Mary Dixon-Woods,et al.  Synthesising qualitative and quantitative evidence: a review of possible methods. , 2005, Journal of health services research & policy.

[31]  E. Tacconelli.  Systematic reviews: CRD's guidance for undertaking reviews in health care , 2010 .

[32]  G. Rücker,et al.  Efficacy and experiences of telephone counselling for informal carers of people with dementia. , 2014, The Cochrane database of systematic reviews.

[33]  Hude Quan,et al.  An administrative data merging solution for dealing with missing data in a clinical registry: adaptation from ICD-9 to ICD-10 , 2008, BMC medical research methodology.

[34]  Lucy Dillon,et al.  Quality in qualitative evaluation: A framework for assessing research evidence. A quality framework. , 2003 .

[35]  J. Cecatti,et al.  Water aerobics II: maternal body composition and perinatal outcomes after a program for low risk pregnant women , 2009, Reproductive health.

[36]  Nancy Cartwright.  Presidential Address: Will This Policy Work for You? Predicting Effectiveness Better: How Philosophy Helps , 2012, Philosophy of Science.

[37]  A. Glassman,et al.  Conditional cash transfers and health: unpacking the causal chain , 2010 .

[38]  Craig Whittington,et al.  Using theory to synthesise evidence from behaviour change interventions: the example of audit and feedback. , 2010, Social science & medicine.

[39]  Sophie Hill,et al.  Taking healthcare interventions from trial to practice , 2010, BMJ : British Medical Journal.

[40]  T. Greenhalgh,et al.  Effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sources , 2005, BMJ : British Medical Journal.

[41]  J. Popay.  Moving beyond floccinaucinihilipilification: enhancing the utility of systematic reviews. , 2005, Journal of clinical epidemiology.

[42]  David R. Jones,et al.  How can systematic reviews incorporate qualitative research? A critical perspective , 2006 .

[43]  Kevin Wheldall.  When will we ever learn? , 2005 .

[44]  Catherine Pope,et al.  Moving Beyond Effectiveness in evidence Synthesis: methodological issues in the synthesis of diverse sources of evidence , 2006 .

[45]  P. Shekelle,et al.  Determining Key Features of Effective Depression Interventions , 2009 .

[46]  H. White,et al.  Farmer Field Schools for Improving Farming Practices and Farmer Outcomes: A Systematic Review. Campbell Systematic Reviews 2014:6. , 2014 .

[47]  Paul Montgomery,et al.  The Oxford Implementation Index: a new tool for incorporating implementation data into systematic reviews and meta-analyses. , 2013, Journal of clinical epidemiology.

[48]  S. Lewin,et al.  Patient Adherence to Tuberculosis Treatment: A Systematic Review of Qualitative Research , 2007, PLoS medicine.

[49]  W. Shadish,et al.  Experimental and Quasi-Experimental Designs for Generalized Causal Inference , 2001 .

[50]  Jenny Donovan,et al.  Evaluating meta-ethnography: a synthesis of qualitative research on lay experiences of diabetes and diabetes care. , 2003, Social science & medicine.

[51]  H Roberts,et al.  Evidence, hierarchies, and typologies: horses for courses , 2003, Journal of epidemiology and community health.

[52]  Laurence S Freedman,et al.  Correction: A simple method for analyzing data from a randomized trial with a missing binary outcome , 2004, BMC Medical Research Methodology.

[53]  Frans L. Leeuw,et al.  Combining Campbell Standards and the Realist Evaluation Approach , 2008 .

[54]  A. Petrosino,et al.  Scared Straight and Other Juvenile Awareness Programs for Preventing Juvenile Delinquency: A Systematic Review of the Randomized Experimental Evidence , 2003, The Cochrane database of systematic reviews.

[55]  P. Knapp,et al.  A systematic review of quantitative and qualitative research on the role and effectiveness of written information available to patients about individual medicines. , 2007, Health technology assessment.

[56]  R. K. Ro D G E R S A N D A N G E L A S Ow D E N,et al.  Testing Methodological Guidance on the Conduct of Narrative Synthesis in Systematic Reviews Effectiveness of Interventions to Promote Smoke Alarm Ownership and Function , 2008 .

[57]  K. Hannes.  Critical Appraisal of Qualitative Research , 2011 .

[58]  Josiane Bonnefoy,et al.  Assessing equity in systematic reviews: realising the recommendations of the Commission on Social Determinants of Health , 2010, BMJ : British Medical Journal.

[59]  M. Petticrew,et al.  Systematic Reviews in the Social Sciences: A Practical Guide , 2005 .

[60]  Catherine Pope,et al.  Resisting medicines: a synthesis of qualitative studies of medicine taking. , 2005, Social science & medicine.

[61]  Mark Engel,et al.  Conducting a meta-ethnography of qualitative literature: Lessons learnt , 2008, BMC medical research methodology.

[62]  S. Ward,et al.  A systematic review and economic evaluation of statins for the prevention of coronary events. , 2007, Health technology assessment.

[63]  M. Scriven.  A Summative Evaluation of RCT Methodology: & An Alternative Approach to Causal Research , 2008, Journal of MultiDisciplinary Evaluation.

[64]  James Thomas,et al.  Bmc Medical Research Methodology Methods for the Synthesis of Qualitative Research: a Critical Review , 2022 .

[65]  Jennie Popay,et al.  Extending systematic reviews to include evidence on implementation: methodological work on a review of community-based initiatives to prevent injuries. , 2006, Social science & medicine.

[66]  A. Petrosino,et al.  Meeting the Challenges of Evidence-Based Policy: The Campbell Collaboration , 2001 .

[67]  P. Tugwell,et al.  Better evidence about wicked issues in tackling health inequities. , 2009, Journal of public health.

[68]  Aron Shlonsky,et al.  Systematic Synthesis of Qualitative Research , 2012 .

[69]  T. Greenhalgh,et al.  Realist review - a new method of systematic review designed for complex policy interventions , 2005, Journal of health services research & policy.

[70]  Mark Petticrew,et al.  Why certain systematic reviews reach uncertain conclusions , 2003, BMJ : British Medical Journal.

[71]  Rebecca Armstrong,et al.  Using logic models to capture complexity in systematic reviews , 2011, Research synthesis methods.

[72]  P. Sandercock,et al.  Framework for design and evaluation of complex interventions to improve health , 2000, BMJ : British Medical Journal.

[73]  Luke Vale,et al.  Evidence-Based Decisions and Economics: Health Care, Social Welfare, Education and Criminal Justice , 2010 .

[74]  Huey-tsyh Chen,et al.  Validity frameworks for outcome evaluation , 2011 .

[75]  Howard White,et al.  Theory-based impact evaluation: principles and practice , 2009 .

[76]  J. Noyes.  Never mind the qualitative feel the depth! The evolving role of qualitative research in Cochrane intervention reviews , 2010 .

[77]  Trisha Greenhalgh,et al.  Realist review to understand the efficacy of school feeding programmes , 2007, British medical journal.

[78]  Michael Woolcock,et al.  Using Mixed Methods in Monitoring and Evaluation: Experiences from International Development , 2010 .

[79]  J. Popay,et al.  Synthesizing Qualitative and Quantitative Health Evidence: A Guide to Methods , 2007 .

[80]  S. Hill,et al.  Notification and support for people exposed to the risk of Creutzfeldt-Jakob disease (CJD) (or other prion diseases) through medical treatment (iatrogenically). , 2011, The Cochrane database of systematic reviews.

[81]  Douglas K Owens,et al.  Challenges in Systematic Reviews: Synthesis of Topics Related to the Delivery, Organization, and Financing of Health Care , 2005, Annals of Internal Medicine.

[82]  Cyrus Samii,et al.  Interventions to promote social cohesion in sub-Saharan Africa , 2010 .

[83]  Andrew V. Dane,et al.  Program integrity in primary and early secondary prevention: are implementation effects out of control? , 1998, Clinical psychology review.

[84]  A. Harden,et al.  Methods for the thematic synthesis of qualitative research in systematic reviews , 2008, BMC medical research methodology.

[85]  W. Hardyman,et al.  Using qualitative research in systematic reviews: Older people’s views of hospital discharge , 2006 .

[86]  S. Øverland,et al.  Electroconvulsive therapy for depression , 2011, Cochrane Database of Systematic Reviews.