Simplifying the selection of evidence synthesis methods to inform environmental decisions: a guide for decision makers and scientists

Achieving evidence-based environmental management requires that decision-makers have access to evidence that can help identify the most effective interventions for their management context. Evidence synthesis supports evidence-based decision-making because it collates, filters and makes sense of a sometimes large and often conflicting evidence-base, potentially yielding new insights. There are many approaches to evidence synthesis. They each have different strengths and weaknesses, making them suited to different purposes, questions and contexts, given particular constraints. To make sense of the wide array of approaches, we outline the important considerations when selecting the most appropriate method for a particular decision context. These include the purpose for the synthesis, the required outcomes, and the multiple constraints within which decision-makers must operate. We then critically assess a spectrum of approaches to evidence synthesis commonly used within environmental management, detailing the characteristics of each that can be used to determine when it is a suitable method. To guide this selection process we provide a decision tree for those commissioning (e.g., decision-makers or stakeholders) or conducting (e.g., scientists) evidence synthesis, which can be used to identify an appropriate method. The decision tree classifies evidence synthesis methods according to whether their purpose is to test or generate hypotheses, the level of resources they require, the level of certainty in the outputs, and the type and scope of the question being addressed. This tool is a major advance because it helps select an appropriate synthesis method based on the multiple constraints that impact the decision. We conclude that there is an approach to evidence synthesis that will suit all management contexts, but that selecting the right approach requires careful consideration of what is fit for purpose.

[1]  Michael J. Stewardson,et al.  An online database and desktop assessment software to simplify systematic reviews in environmental science , 2015, Environ. Model. Softw..

[2]  Veronica A. J. Doerr,et al.  Maximizing the value of systematic reviews in ecology when data or resources are limited , 2015 .

[3]  I. Boyd,et al.  How to increase the potential policy impact of environmental science research , 2015, Environmental Sciences Europe.

[4]  Gary Banks,et al.  Evidence-Based Policy Making: What is It? How Do We Get It? , 2009, World Scientific Reference on Asia-Pacific Trade Policies.

[5]  S. Golder,et al.  The effectiveness and cost-effectiveness of prophylactic removal of wisdom teeth. , 2000, Health technology assessment.

[6]  Jessica Gurevitch,et al.  Transparency in Ecology and Evolution: Real Problems, Real Solutions. , 2016, Trends in ecology & evolution.

[7]  A. Pullin,et al.  Support for decision making in conservation practice: an evidence-based approach , 2003 .

[8]  A. B. Hill The Environment and Disease: Association or Causation? , 1965, Proceedings of the Royal Society of Medicine.

[9]  J. A. Wardekker,et al.  Uncertainty Communication : Issues and good practice , 2007 .

[10]  M. Schwartz,et al.  Achieving Conservation Science that Bridges the Knowledge–Action Boundary , 2013, Conservation biology : the journal of the Society for Conservation Biology.

[11]  Neal R Haddaway,et al.  Making literature reviews more reliable through application of lessons from systematic reviews , 2015, Conservation biology : the journal of the Society for Conservation Biology.

[12]  R. Aarde,et al.  Science and elephant management decisions in South Africa , 2011 .

[13]  I. Nisbet Bird Conservation: Global Evidence for the Effects of Interventions , 2013 .

[14]  Paul Wang,et al.  Evaluations with impact: decision-focused impact evaluation as a practical policymaking tool , 2015 .

[15]  D. Cook,et al.  Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? , 1998, The Lancet.

[16]  Carsten Nesshöver,et al.  Biodiversity knowledge synthesis at the European scale: actors and steps , 2016, Biodiversity and Conservation.

[17]  J. Webb,et al.  Environmental Flows Can Reduce the Encroachment of Terrestrial Vegetation into River Channels: A Systematic Literature Review , 2013, Environmental Management.

[18]  Susan J. Nichols,et al.  Multiple Lines and Levels of Evidence for Detecting Ecological Responses to Management Intervention , 2005 .

[19]  Andrew S. Pullin,et al.  Effectiveness in Conservation Practice: Pointers from Medicine and Public Health , 2001 .

[20]  William J. Sutherland,et al.  What works in conservation? Using expert assessment of summarised evidence to identify practices that enhance natural pest control in agriculture , 2016, Biodiversity and Conservation.

[21]  Rebecca K. Smith,et al.  Is nest predator exclusion an effective strategy for enhancing bird populations , 2011 .

[22]  Hanne Foss Hansen,et al.  The Evidence Movement , 2009 .

[23]  J. Webb,et al.  Challenges for evidence-based environmental management: what is acceptable and sufficient evidence of causation? , 2017, Freshwater Science.

[24]  George Davey Smith,et al.  meta-analysis bias in location and selection of studies , 1998 .

[25]  N. Poff,et al.  A General Approach to Predicting Ecological Responses to Environmental Flows: Making Best Use of the Literature, Expert Knowledge, and Monitoring Data , 2015 .

[26]  Jessica Gurevitch,et al.  THE META‐ANALYSIS OF RESPONSE RATIOS IN EXPERIMENTAL ECOLOGY , 1999 .

[27]  M. Petticrew,et al.  Systematic Reviews in the Social Sciences: A Practical Guide , 2005 .

[28]  S. Norton,et al.  Weaving common threads in environmental causal assessment methods: toward an ideal method for rapid evidence synthesis , 2017, Freshwater Science.

[29]  S. Lewis,et al.  Meta-analysis and The Cochrane Collaboration: 20 years of the Cochrane Statistical Methods Group , 2013, Systematic Reviews.

[30]  Richard Margoluis,et al.  Using conceptual models as a planning and evaluation tool in conservation. , 2009, Evaluation and program planning.

[31]  Rebecca K. Smith,et al.  What Works in Conservation: 2017 , 2017 .

[32]  A. Pullin,et al.  Doing more good than harm: building an evidence-base for conservation and environmental management. , 2009 .

[33]  A. Pullin,et al.  Guidelines for Systematic Review in Conservation and Environmental Management , 2006, Conservation biology : the journal of the Society for Conservation Biology.

[34]  Andrew S Pullin,et al.  Effectiveness of Management Interventions to Control Invasion by Rhododendron ponticum , 2006, Environmental management.

[35]  Susan B. Norton,et al.  Conceptual model diagrams as evidence scaffolds for environmental assessment and management , 2017, Freshwater Science.

[36]  Gurutzeta Guillera-Arroita,et al.  When do we need more data? A primer on calculating the value of information for applied ecologists , 2015 .

[37]  Ruth Garside,et al.  What are the effects of nature conservation on human well-being? A systematic map of empirical evidence from developing countries , 2016, Environmental Evidence.

[38]  Heli Saarikoski,et al.  Selecting appropriate methods of knowledge synthesis to inform biodiversity policy , 2016, Biodiversity and Conservation.

[39]  R. Cousens,et al.  The importance of seasonal flow timing for riparian vegetation dynamics: a systematic review using causal criteria analysis , 2011 .

[40]  Michel J. Kaiser,et al.  Evaluating and improving the reliability of evidence syntheses in conservation and environmental science: A methodology , 2014 .

[41]  Ross J. Harris,et al.  How much of the data published in observational studies of the association between diet and prostate or bladder cancer is usable for meta-analysis? , 2008, American journal of epidemiology.

[42]  Andrew S. Pullin,et al.  Are review articles a reliable source of evidence to support conservation and environmental management? A comparison with medicine , 2006 .

[43]  G. Rees,et al.  What is the evidence for glacial shrinkage across the Himalayas , 2013 .

[44]  D. Faith,et al.  Monitoring Ecological Impacts: Concepts and Practice in Flowing Waters , 2002 .

[45]  M Susser,et al.  What is a cause and how do we know one? A grammar for pragmatic epidemiology. , 1991, American journal of epidemiology.

[46]  W. Sutherland,et al.  The need for evidence-based conservation. , 2004, Trends in ecology & evolution.

[47]  David Moher,et al.  Evidence summaries: the evolution of a rapid review approach , 2012, Systematic Reviews.

[48]  H. Possingham,et al.  Contribution of Systematic Reviews to Management Decisions , 2013, Conservation biology : the journal of the Society for Conservation Biology.

[49]  William J Sutherland,et al.  Organising evidence for environmental management decisions: a '4S' hierarchy. , 2014, Trends in ecology & evolution.

[50]  S. Adams Assessing cause and effect of multiple stressors on marine systems. , 2005, Marine pollution bulletin.

[51]  Craig Fischenich The application of conceptual models to ecosystem restoration , 2008 .

[52]  Rebecca Armstrong,et al.  Using logic models to capture complexity in systematic reviews , 2011, Research synthesis methods.

[53]  Evan Harrison,et al.  Analyzing cause and effect in environmental assessments: using weighted evidence from the literature , 2011, Freshwater Science.

[54]  David Hailey,et al.  Rapid reviews versus full systematic reviews: An inventory of current methods and practice in health technology assessment , 2008, International Journal of Technology Assessment in Health Care.

[55]  Alex J. Sutton,et al.  Publication and related biases: a review , 2000 .

[56]  Adrian C. Newton,et al.  Bayesian Belief Networks as a tool for evidence-based conservation management , 2007 .

[57]  J. Watson,et al.  Achieving Open Access to Conservation Science , 2014, Conservation biology : the journal of the Society for Conservation Biology.

[58]  G. Arnqvist,et al.  Meta-analysis: synthesizing research findings in ecology and evolution. , 1995, Trends in ecology & evolution.

[59]  Rebecca K. Smith,et al.  Farmland Conservation: Evidence for the effects of interventions in northern and western Europe , 2014 .

[60]  Mitchell Pavao-Zuckerman,et al.  Conceptual Models as Tools for Communication Across Disciplines , 2003 .

[61]  Duncan N. L. Menge,et al.  Gauging the impact of meta-analysis on ecology , 2012, Evolutionary Ecology.

[62]  David B. Lindenmayer,et al.  Can methods applied in medicine be used to summarize and disseminate conservation research? , 2004, Environmental Conservation.

[63]  C. Mulrow,et al.  Systematic Reviews: Rationale for systematic reviews , 1994 .

[64]  D. Gough,et al.  An Introduction to Systematic Reviews , 2017 .

[65]  Neal R. Haddaway,et al.  A methodology for systematic mapping in environmental sciences , 2016, Environmental Evidence.

[66]  Gary Bilotta,et al.  On the use of systematic reviews to inform environmental policies , 2014 .

[67]  W. Sutherland,et al.  The effect of scientific evidence on conservation practitioners’ management decisions , 2014, Conservation biology : the journal of the Society for Conservation Biology.

[68]  D. Weed,et al.  On the use of causal criteria. , 1997, International journal of epidemiology.

[69]  Evidence-based conservation and evidence-informed policy: a response to Adams & Sandbrook , 2013, Oryx.

[70]  S. Norton,et al.  Digital repository of associations between environmental variables: A new resource to facilitate knowledge synthesis , 2015 .