Time to rethink the systematic review catechism? Moving from ‘what works’ to ‘what happens’

Systematic review methods are developing rapidly, and most researchers would recognise their key methodological aspects, such as a closely focussed question, a comprehensive search, and a focus on synthesising ‘stronger’ rather than ‘weaker’ evidence. However, it may be helpful to question some of these underlying principles, because while they work well for simpler review questions, they may result in overly narrow approaches to more complex questions and interventions. This commentary discusses some core principles of systematic reviews, and how they may require further rethinking, particularly as reviewers turn their attention to increasingly complex issues, where a Bayesian perspective on evidence synthesis, which would aim to assemble evidence - of different types, if necessary - in order to inform decisions’, may be more productive than the ‘traditional’ systematic review model. Among areas identified for future research are the examination of publication bias in qualitative research; research on the efficiency and potential biases of comprehensive searches in different disciplines; and the use of Bayesian methods in evidence synthesis. The incorporation of a systems perspective into systematic reviews is also an area which needs rapid development.

[1]  J. Higgins Cochrane handbook for systematic reviews of interventions. Version 5.1.0 [updated March 2011]. The Cochrane Collaboration , 2011 .

[2]  M. Sculpher,et al.  Using Value of Information Analysis to Prioritise Health Research , 2012, PharmacoEconomics.

[3]  J. Popay Qualitative research and the epidemiological imagination: a vital relationship. , 2003, Gaceta Sanitaria.

[4]  C. Pope,et al.  Using meta-ethnography to synthesise qualitative research , 2006 .

[5]  S. Hopewell,et al.  Cochrane methods - twenty years experience in developing systematic review methods , 2013, Systematic Reviews.

[6]  Andrew Booth,et al.  How much searching is enough? Comprehensive versus optimal retrieval for technology assessments , 2010, International Journal of Technology Assessment in Health Care.

[7]  K. D'angremond,et al.  Use of Theory , 1995 .

[8]  Margarete Sandelowski,et al.  Mapping the Mixed Methods–Mixed Research Synthesis Terrain , 2012, Journal of mixed methods research.

[9]  David Moher,et al.  Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews , 2007, BMC medical research methodology.

[10]  Jillian Whelan,et al.  Cochrane update: Predicting sustainability of intervention effects in public health evidence: identifying key elements to provide guidance. , 2014, Journal of public health.

[11]  M. Petticrew,et al.  Housing Improvements for Health and Associated Socio‐Economic Outcomes: A Systematic Review , 2013 .

[12]  David Ogilvie,et al.  Systematic reviews of health effects of social interventions: 2. Best available evidence: how low should you go? , 2005, Journal of Epidemiology and Community Health.

[13]  Alex J. Sutton,et al.  Integrative approaches to qualitative and quantitative evidence , 2004 .

[14]  P. Tugwell,et al.  Non‐randomized studies as a source of complementary, sequential or replacement evidence for randomized controlled trials in systematic reviews on the effects of interventions , 2013, Research synthesis methods.

[15]  J. Popay,et al.  Guidance on the conduct of narrative synthesis in sytematic reviews , 2006 .

[16]  M. Petticrew,et al.  Understanding the Psychosocial Impacts of Housing Type: Qualitative Evidence from a Housing and Regeneration Intervention , 2011 .

[17]  D. Moher The problem of duplicate systematic reviews , 2013, BMJ.

[18]  M. Kelly,et al.  The appraisal of public health interventions: the use of theory. , 2015, Journal of public health.

[19]  Alan Shiell,et al.  Theorising Interventions as Events in Systems , 2009, American journal of community psychology.

[20]  Andrew Booth,et al.  Celebrating methodological challenges and changes: reflecting on the emergence and importance of the role of qualitative evidence in Cochrane reviews , 2013, Systematic Reviews.

[21]  D. Moher,et al.  PROSPERO at one year: an evaluation of its utility , 2013, Systematic Reviews.

[22]  J. Sterne,et al.  How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study. , 2003, Health technology assessment.

[23]  Guy M. Goodwin,et al.  Introduction to Systematic Reviews , 2004, Journal of psychopharmacology.

[24]  M. Petticrew,et al.  Systematic Reviews in the Social Sciences: A Practical Guide , 2005 .

[25]  Ray Pawson,et al.  RAMESES publication standards: meta-narrative reviews , 2013, BMC Medicine.

[26]  Ray Pawson,et al.  RAMESES publication standards: realist syntheses , 2013, BMC Medicine.

[27]  Jennie Popay,et al.  Guidance on the conduct of narrative synthesis in systematic Reviews. A Product from the ESRC Methods Programme. Version 1 , 2006 .

[28]  Sandy Oliver,et al.  Integrating qualitative research with trials in systematic reviews , 2004, BMJ : British Medical Journal.

[29]  Keith R Abrams,et al.  Factors affecting uptake of childhood immunisation: a Bayesian synthesis of qualitative and quantitative evidence , 2002, The Lancet.

[30]  Trisha Greenhalgh,et al.  Protocol - realist and meta-narrative evidence synthesis: Evolving Standards (RAMESES) , 2011, BMC medical research methodology.

[31]  M. Petticrew,et al.  Housing improvements for health and associated socio-economic outcomes. , 2013, The Cochrane database of systematic reviews.

[32]  M. Petticrew,et al.  Systematic Reviews in the Social Sciences , 2006 .

[33]  M. Petticrew,et al.  Promoting walking and cycling as an alternative to using cars: systematic review , 2004, BMJ : British Medical Journal.

[34]  M. Petticrew Public Health Evaluation: Epistemological Challenges to Evidence Production and Use. , 2013 .

[35]  Ruth Garside,et al.  The role of systematic reviews of qualitative evidence in evaluating interventions: a case study , 2012, Research synthesis methods.