Why and how to combine evidence in environmental assessments: weighing evidence and building cases.

All types of environmental decisions benefit from assessments that assemble and analyze diverse evidence. The diversity of that evidence creates complexities that can be managed using an explicit, well-designed process. We suggest two adaptations from the legal lexicon, weight of evidence and building a case. When weighing evidence, weights are assigned to each piece of evidence, and then the body of evidence is weighed in favor of each hypothesis by amassing the weights. Finally, the total weights of evidence for the alternative hypotheses are compared to determine which alternative has the preponderance of evidence in its favor. When building a case, pieces of evidence are organized to show relationships among multiple hypotheses or complex interactions among agents, events, or processes. We provide processes for weighing evidence and building a case and illustrate both approaches in a case study involving the decline of a kit fox population. The general approach presented here is flexible, transparent, and defensible. During its development, it has been applied to risk assessments for contaminated sites and to causal assessments in aquatic and terrestrial systems. It is intended to balance the need for rigor and discipline with the need for sufficient flexibility to accept all relevant evidence and generate creative solutions to difficult environmental problems.

[1]  I. Good A CAUSAL CALCULUS (I)* , 1961, The British Journal for the Philosophy of Science.

[2]  M M Mumtaz,et al.  A Weight-of-Evidence Approach for Assessing Interactions in Chemical Mixtures , 1992, Toxicology and industrial health.

[3]  William J. Mitsch,et al.  Enhancing the Ecological Risk Assessment Process , 2008, Integrated environmental assessment and management.

[4]  Michael C. Newman,et al.  Coastal and estuarine ecological risk assessment: the need for a more formal approach to stressor identification , 2007 .

[5]  Richard H. Norris,et al.  DEVELOPMENT AND EVALUATION OF PREDICTIVE MODELS FOR MEASURING THE BIOLOGICAL INTEGRITY OF STREAMS , 2000 .

[6]  Dick de Zwart,et al.  Diagnosis of Ecosystem Impairment in a Multiple-Stress Context—How to Formulate Effective River Basin Management Plans , 2009, Integrated environmental assessment and management.

[7]  B. Brook,et al.  Modelling to forestall extinction of Australian tropical birds , 2007, Journal of Ornithology.

[8]  Gary S. Lawrence,et al.  Weight-of-Evidence Issues and Frameworks for Sediment Quality (And Other) Assessments , 2002 .

[9]  Peter M Chapman,et al.  Design and Application of a Transparent and Scalable Weight-of-Evidence Framework: An Example From Wabamun Lake, Alberta, Canada , 2007, Integrated environmental assessment and management.

[10]  H. Akaike INFORMATION THEORY AS AN EXTENSION OF THE MAXIMUM LIKELIHOOD , 1973 .

[11]  Valery E. Forbes,et al.  A Weight-of-Evidence Framework for Assessing Sediment (Or Other) Contamination: Improving Certainty in the Decision-Making Process , 2002 .

[12]  Valery E. Forbes,et al.  Applying Weight-of-Evidence in Retrospective Ecological Risk Assessment When Quantitative Data Are Limited , 2002 .

[13]  D. Weed,et al.  Interpreting epidemiological evidence: how meta-analysis and causal inference methods are related. , 2000, International journal of epidemiology.

[14]  U. Epa Guidelines for carcinogen risk assessment , 1986 .

[15]  Tony Delamothe,et al.  Open access publishing takes off , 2004, BMJ : British Medical Journal.

[16]  S. Lazic,et al.  Model based Inference in the Life Sciences: a Primer on Evidence , 2011 .

[17]  Peter M. Chapman,et al.  A Sediment Quality Triad: Measures of sediment contamination, toxicity and infaunal community composition in Puget Sound , 1985 .

[18]  I. Good Good Thinking: The Foundations of Probability and Its Applications , 1983 .

[19]  A Field-based Aquatic Life Benchmark for Conductivity in Central Appalachian Streams , 2010 .

[20]  Vern R. Walker RISK CHARACTERIZATION AND THE WEIGHT OF EVIDENCE : ADAPTING GATEKEEPING CONCEPTS FROM THE COURTS , 1996 .

[21]  Peter M Chapman,et al.  Traditional ecological knowledge (TEK) and scientific weight of evidence determinations. , 2007, Marine pollution bulletin.

[22]  Igor Linkov,et al.  Weight-of-evidence evaluation in environmental assessment: review of qualitative and quantitative approaches. , 2009, The Science of the total environment.

[23]  Rebecca A. Efroymson,et al.  Ecological risk assessment in a large river‐reservoir: 2. Fish community , 1999 .

[24]  P. Chapman,et al.  The importance of benthos in weight of evidence sediment assessments--a case study. , 2008, The Science of the total environment.

[25]  Michele Tarsilla Cochrane Handbook for Systematic Reviews of Interventions , 2010, Journal of MultiDisciplinary Evaluation.

[26]  N McKoy,et al.  Systems to rate the strength of scientific evidence. , 2002, Evidence report/technology assessment.

[27]  Sheldon Krimsky,et al.  The weight of scientific evidence in policy and law. , 2005, American journal of public health.

[28]  Elizabeth Southerland,et al.  Technical Support Document for Water Quality-based Toxics Control , 1991 .

[29]  A R Jadad,et al.  Assessing the quality of reports of randomized clinical trials: is blinding necessary? , 1996, Controlled clinical trials.

[30]  Jonathan I Levy,et al.  Science and Decisions: Advancing Risk Assessment , 2010, Risk analysis : an official publication of the Society for Risk Analysis.

[31]  K. Lohr Rating the strength of scientific evidence: relevance for quality improvement programs. , 2004, International journal for quality in health care : journal of the International Society for Quality in Health Care.

[32]  J. Gold,et al.  Banburismus and the Brain Decoding the Relationship between Sensory Stimuli, Decisions, and Reward , 2002, Neuron.

[33]  P. Chapman,et al.  Selenium effects: A weight‐of‐evidence approach , 2007, Integrated environmental assessment and management.

[34]  Glenn W Suter,et al.  Using Field Data and Weight of Evidence to Develop Water Quality Criteria , 2008, Integrated environmental assessment and management.

[35]  D. Weed Weight of Evidence: A Review of Concept and Methods , 2005, Risk analysis : an official publication of the Society for Risk Analysis.

[36]  G A Fox,et al.  Practical causal inference for ecoepidemiologists. , 1991, Journal of toxicology and environmental health.

[37]  Ruth N Hull,et al.  Sequential Analysis of Lines of Evidence—An Advanced Weight-of-Evidence Approach for Ecological Risk Assessment , 2006, Integrated environmental assessment and management.

[38]  Glenn W. Suter,et al.  Causal Characteristics for Ecoepidemiology , 2010 .

[39]  Peter M. Chapman,et al.  The sediment quality triad approach to determining pollution-induced degradation , 1990 .

[40]  Rebecca A. Efroymson,et al.  Ecological Risk Assessment for Contaminated Sites , 2000 .

[41]  Peter M. Chapman,et al.  THE SEDIMENT QUALITY TRIAD: THEN, NOW AND TOMORROW , 2000 .

[42]  Glenn W Suter,et al.  A Framework for Fully Integrating Environmental Assessment , 2008, Environmental management.

[43]  Susan M. Cormier,et al.  Causal Assessment of Biological Impairment in the Little Floyd River, Iowa, USA , 2010 .

[44]  Susan M. Cormier,et al.  Assessment of Probable Causes of Reduced Aquatic Life in the Touchet River, Washington, USA , 2010 .

[45]  S. Cormier,et al.  When is a Formal Assessment Process Worthwhile? , 2010 .

[46]  Alexandre Santos Aguiar,et al.  Weight of evidence , 2008 .

[47]  P. Burkhardt-Holm,et al.  Application of the weight-of-evidence approach to assess the decline of brown trout (Salmo trutta) in Swiss rivers , 2007, Aquatic Sciences.

[48]  P.,et al.  Weight-of-Evidence ( WOE ) : Quantitative Estimation of Probability of Impairment for Individual and Multiple Lines of Evidence , 2006 .

[49]  M Susser,et al.  Rules of inference in epidemiology. , 1986, Regulatory toxicology and pharmacology : RTP.

[50]  G. Guyatt,et al.  Grading quality of evidence and strength of recommendations , 2004, BMJ : British Medical Journal.

[51]  Chris O. Yoder,et al.  The Role of Biological Indicators in a State Water Quality Management Process , 1998 .

[52]  Glenn W. Suter,et al.  Ecological risk assessment , 2006 .

[53]  D. Mitchell,et al.  Special report of the Massachusetts weight‐of‐evidence workgroup A weight‐of‐evidence approach for evaluating ecological risks , 1996 .

[54]  Ord,et al.  Guidelines for Ecological Risk Assessment , 2014 .

[55]  Simon Gates,et al.  Review of methodology of quantitative reviews using meta‐analysis in ecology , 2002 .

[56]  Glenn W Suter,et al.  A methodology for inferring the causes of observed impairments in aquatic ecosystems , 2002, Environmental toxicology and chemistry.

[57]  R. Hilborn,et al.  The Ecological Detective: Confronting Models with Data , 1997 .

[58]  J. Popay,et al.  Synthesizing Qualitative and Quantitative Health Evidence: A Guide to Methods , 2007 .

[59]  Rebecca A. Efroymson,et al.  Ecological risk assessment in a large river‐reservoir: 3. Benthic invertebrates , 1999 .

[60]  L. Cavalli-Sforza,et al.  Genes, peoples, and languages. , 1997, Proceedings of the National Academy of Sciences of the United States of America.

[61]  D Moher,et al.  The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomized trials. , 2001, Annals of internal medicine.

[62]  John Jensen,et al.  A “Weight-of-Evidence” Approach for the Integration of Environmental “Triad” Data to Assess Ecological Risk and Biological Vulnerability , 2008, Integrated environmental assessment and management.

[63]  Andrea Critto,et al.  Decision Support Systems for Risk-Based Management of Contaminated Sites , 2009 .