Developing and Optimising the Use of Logic Models in Systematic Reviews: Exploring Practice and Good Practice in the Use of Programme Theory in Reviews

Background Logic models are becoming an increasingly common feature of systematic reviews, as is the use of programme theory more generally in systematic reviewing. Logic models offer a framework to help reviewers to ‘think’ conceptually at various points during the review, and can be a useful tool in defining study inclusion and exclusion criteria, guiding the search strategy, identifying relevant outcomes, identifying mediating and moderating factors, and communicating review findings. Methods and Findings In this paper we critique the use of logic models in systematic reviews and protocols drawn from two databases representing reviews of health interventions and international development interventions. Programme theory featured only in a minority of the reviews and protocols included. Despite drawing from different disciplinary traditions, reviews and protocols from both sources shared several limitations in their use of logic models and theories of change, and these were used almost unanimously to solely depict pictorially the way in which the intervention worked. Logic models and theories of change were consequently rarely used to communicate the findings of the review. Conclusions Logic models have the potential to be an aid integral throughout the systematic reviewing process. The absence of good practice around their use and development may be one reason for the apparent limited utility of logic models in many existing systematic reviews. These concerns are addressed in the second half of this paper, where we offer a set of principles in the use of logic models and an example of how we constructed a logic model for a review of school-based asthma interventions.

[1]  Claire Stansfield,et al.  Finding relevant studies , 2012 .

[2]  Andrew Pettigrew,et al.  Scholarship with Impact , 2011 .

[3]  Sandy Oliver,et al.  Stakeholder perspectives and participation in reviews , 2012 .

[4]  U. Lynch,et al.  What is the evidence that the establishment or use of community accountability mechanisms and processes improves inclusive service delivery by governments, donors and NGOs to communities? , 2013 .

[5]  C. Abraham,et al.  Effective techniques in healthy eating and physical activity interventions: a meta-regression. , 2009, Health psychology : official journal of the Division of Health Psychology, American Psychological Association.

[6]  Kerrie Mengersen,et al.  Systematic review protocol: The impact of policing interventions on violent crime in developing countries , 2013 .

[7]  P. Murthy,et al.  PROTOCOL: The Effects Of Economic Self‐Help Group Programs On Women's Empowerment: A Systematic Review , 2013 .

[8]  Steven Cummins,et al.  ‘Dark logic’: theorising the harmful consequences of public health interventions , 2014, Journal of Epidemiology & Community Health.

[9]  Peter Tugwell,et al.  How to do a good systematic review of effects in international development: a tool kit , 2012 .

[10]  Michael D Cabana,et al.  Do School-Based Asthma Education Programs Improve Self-Management and Health Outcomes? , 2009, Pediatrics.

[11]  Stephen Porder,et al.  Bridge over a philosophical divide , 2006 .

[12]  Sandy Oliver,et al.  Introducing Systematic Reviews , 2019, Conducting Systematic Reviews in Sport, Exercise, and Physical Activity.

[13]  Geeta Kingdon,et al.  Protocol - Are contract and para-teachers a cost effective intervention to address teachershortages and improve learning outcomes? , 2012 .

[14]  Jane Barlow,et al.  Psychological interventions to prevent recurrence of emotional abuse of children by their parents , 2013 .

[15]  Josip Car,et al.  Systematic Review. Effectiveness of interventions to strengthen national health service delivery on coverage, access, quality, and equity in the use of health services in low and lower middle income countries. , 2012 .

[16]  S. Oliver,et al.  Community engagement to reduce inequalities in health: a systematic review, meta-analysis and economic analysis , 2013 .

[17]  David A. Robalino,et al.  Interventions to Improve the Labour Market Outcomes of Youth: A Systematic Review of Training, Entrepreneurship Promotion, Employment Services and Subsidized Employment Interventions , 2017 .

[18]  Jacqui Webster,et al.  Iodine fortification of foods and condiments, other than salt, for preventing iodine deficiency disorders , 2013 .

[19]  Damien John Williams,et al.  Universal community‐based social development interventions for preventing community violence by young people 12 to 18 years of age , 2017 .

[20]  Sandy Oliver,et al.  Damned if you do, damned if you don't: subgroup analysis and equity , 2011, Journal of Epidemiology & Community Health.

[21]  E. Waters,et al.  The WHO Health Promoting School framework for improving the health and well-being of students and their academic achievement. , 2014, The Cochrane database of systematic reviews.

[22]  Barbara M. Bakker,et al.  Interventions for promoting reintegration and reducing harmful behaviour and lifestyles in street-connected children and young people. , 2016, The Cochrane database of systematic reviews.

[23]  Miguel Niño-Zarazúa,et al.  WIDER Working Paper No . 2013 / 087 What do we know about non-clinical interventions for preventable and treatable childhood diseases in developing countries ? , 2013 .

[24]  David Evans,et al.  School-based asthma programs. , 2009, The Journal of allergy and clinical immunology.

[25]  Craig Whittington,et al.  Using theory to synthesise evidence from behaviour change interventions: the example of audit and feedback. , 2010, Social science & medicine.

[26]  Robin Christensen,et al.  Interventions to improve access to cataract surgical services and their impact on equity in low- and middle-income countries. , 2014, The Cochrane database of systematic reviews.

[27]  Geeta Kingdon,et al.  Are contract and para-teachers a cost effective intervention to address teacher shortages and improve learning outcomes ? PROTOCOL , 2012 .

[28]  Sandy Oliver,et al.  Psychosocial interventions for supporting women to stop smoking in pregnancy. , 2013, The Cochrane database of systematic reviews.

[29]  Robert Fletcher,et al.  Deworming and adjuvant interventions for improving the developmental health and well-being of children in low-and middle-income countries : a systematic review and network meta-analysis , 2015 .

[30]  C. Sreeramareddy,et al.  Decentralised versus centralised governance of health services , 2013, Cochrane Database of Systematic Reviews.

[31]  Richard Smith,et al.  Measuring the social impact of research , 2001, BMJ : British Medical Journal.

[32]  Robert H. Fletcher,et al.  Deworming and adjuvant interventions for improving the developmental health and well‐being of children in low‐ and middle‐income countries: a systematic review and network meta‐analysis , 2013 .

[33]  Huey-tsyh Chen Theory-driven evaluations , 1990 .

[34]  Edward H. Yelin,et al.  Do School-Based Asthma Education Programs Improve Self-Management and Health Outcomes?. , 2009, ATS 2009.

[35]  Sandy Oliver,et al.  The politics of evidence and methodology: lessons from the EPPI-Centre , 2005 .

[36]  Eva Rehfuess,et al.  Slum upgrading strategies involving physical environment and infrastructure interventions and their effects on health and socio-economic outcomes. , 2013, The Cochrane database of systematic reviews.

[37]  Geraldine Macdonald,et al.  Interventions to build resilience in children of problem drinkers , 2014 .

[38]  Janice Tripney,et al.  Technical and Vocational Education and Training (TVET) Interventions to Improve the Employability and Employment of Young People in Low‐ and Middle‐Income Countries: A Systematic Review , 2013 .

[39]  Rebecca Armstrong,et al.  Using logic models to capture complexity in systematic reviews , 2011, Research synthesis methods.

[40]  P Davies,et al.  The Campbell Collaboration , 2001, BMJ : British Medical Journal.

[41]  Astrid Brousselle,et al.  Program theory evaluation: Logic analysis. , 2011, Evaluation and program planning.

[42]  D. Gough,et al.  An Introduction to Systematic Reviews , 2017 .

[43]  N. Clark,et al.  Effects of educational interventions for self management of asthma in children and adolescents: systematic review and meta-analysis , 2003, BMJ : British Medical Journal.

[44]  T.C.B. Chancellor,et al.  A systematic review on the impacts of capacity strengthening of agricultural research systems for development and the conditions of success. , 2013 .

[45]  Candida Fenton,et al.  Considering methodological options for reviews of theory: illustrated by a review of theories linking income and health , 2014, Systematic Reviews.

[46]  Linnet Taylor,et al.  The Impact of Business Support Services for Small and Medium Enterprises on Firm Performance in Low- and Middle-Income Countries : A Systematic Review , 2016 .

[47]  Elizabeth Goyder,et al.  Using logic model methods in systematic review synthesis: describing complex pathways in referral management interventions , 2014, BMC Medical Research Methodology.

[48]  Marina Davoli,et al.  Interventions for reducing adverse health effects of high temperature and heatwaves , 2014, Cochrane Database of Systematic Reviews.

[49]  David Hevey,et al.  Educational programmes for primary prevention of skin cancer (Protocol) , 2014 .

[50]  Jacob Burns,et al.  Interventions to reduce ambient particulate matter air pollution and their effect on health , 2014 .

[51]  Rona Campbell,et al.  Exploring the feasibility of theory synthesis: A worked example in the field of health related risk-taking , 2015, Social science & medicine.

[52]  Miklas Scholz,et al.  How effective are reedbeds, ponds, restored and constructed wetlands at retaining nitrogen, phosphorus and suspended sediment from agricultural pollution in England? , 2013, Environmental Evidence.

[53]  Esther Coren,et al.  Response by authors. , 2013 .

[54]  Ray Wu,et al.  Making an impact , 2004, Nature.

[55]  X. Cirera,et al.  The Impact of Export Processing Zones on Employment , Wages and Labour Conditions in Developing Countries , 2013 .

[56]  Laurenz Langer,et al.  What are the impacts of urban agriculture programs on food security in low and middle-income countries? , 2013, Environmental Evidence.

[57]  James Sumberg,et al.  Under What Circumstances and Conditions Does Adoption of Technology Result in Increased Agricultural Productivity? A Systematic Review , 2013 .

[58]  Kerrie Mengersen,et al.  The impact of policing interventions on interpersonal violent crime in developing countries (Protocol) , 2013 .

[59]  Ursula Giedion,et al.  The Impact of Universal Coverage Schemes in the Developing World : A Review of the Existing Evidence , 2013 .

[60]  Glyn Williams,et al.  The disciplining effects of impact evaluation practices: negotiating the pressures of impact within an ESRC–DFID project , 2012 .

[61]  M. Kelly,et al.  Synthesizing diverse evidence: the use of primary qualitative data analysis methods and logic models in public health reviews. , 2010, Public health.

[62]  Claire Glenton,et al.  Barriers and facilitators to the implementation of lay health worker programmes to improve access to maternal and child health: qualitative evidence synthesis. , 2013, The Cochrane database of systematic reviews.

[63]  Ian B Stewart,et al.  Cochrane review: whole‐body cryotherapy (extreme cold air exposure) for preventing and treating muscle soreness after exercise in adults , 2016, Journal of evidence-based medicine.

[64]  Guillermo Cruces,et al.  Community Monitoring to Curb Corruption and Increase Efficiency in Service Delivery: Evidence from Low Income Communities. , 2013 .

[65]  P. Halligan,et al.  Making an impact. , 2012, Cognitive neuropsychiatry.

[66]  Pamela Collins,et al.  Roles of the state asthma program in implementing multicomponent, school-based asthma interventions. , 2013, The Journal of school health.

[67]  P. Kearney,et al.  The effect of moderate gestational alcohol consumption during pregnancy on speech and language outcomes in children: a systematic review , 2014, Systematic Reviews.

[68]  Patricia J. Rogers,et al.  Purposeful Program Theory: Effective Use of Theories of Change and Logic Models , 2011 .

[69]  Kathryn M McDonald,et al.  Evaluating the state of quality-improvement science through evidence synthesis: insights from the closing the quality gap series. , 2013, The Permanente journal.

[70]  Anke Rohwer,et al.  Logic model templates for systematic reviews of complex health interventions , 2013 .

[71]  Josip Car,et al.  Effectiveness of interventions to strengthen national health service delivery on coverage, access, quality, and equity in the use of health services in low and lower middle income countries , 2012 .

[72]  Ian B Stewart,et al.  Whole-body cryotherapy (extreme cold air exposure) for preventing and treating muscle soreness after exercise in adults. , 2015, The Cochrane database of systematic reviews.

[73]  D. Gough,et al.  Commonality and diversity in reviews , 2012 .

[74]  Janice Tripney,et al.  Protocol for a Systematic Review: Interventions to Improve the Labour Market Situation of Adults with Physical and/or Sensory Disabilities in Low‐ and Middle‐Income Countries , 2013 .