Evaluating reliability and risk of bias of in vivo animal data for risk assessment of chemicals – Exploring the use of the SciRAP tool in a systematic review context

Within the field of health risk assessment, it is essential that evaluations of reliability or validity of toxicity data are conducted with structure and transparency. To this end, different tools for evaluating toxicity studies have been developed by different groups and organizations, for different specific purposes. The Science in Risk Assessment and Policy (SciRAP) tool was developed for use in the regulatory health risk assessment of chemicals and to promote structured and transparent evaluation of study reliability within European regulatory frameworks. As such, the SciRAP tool is not specifically tailored for use in a systematic review context. However, in light of the current movement towards applying systematic review in the field of environmental health and chemical assessments and European chemicals regulation, we were interested in exploring how SciRAP could be applied in such a context. To achieve this, the scope of the SciRAP tool was first compared to two tools developed based on systematic review principles at the US Environmental Protection Agency's IRIS program and the National Toxicology Program's Office of Health Assessment and Translation (OHAT). Next, the SciRAP and IRIS tools were both applied in a case study to evaluate the same nine in vivo animal studies and the resulting evaluations were compared. The SciRAP tool was found to address the majority of the elements included for study evaluation in the OHAT and IRIS tools. In the case study, no major differences were found in the conclusions drawn when using SciRAP or IRIS tools. However, future developments to bring the SciRAP tool more in line with systematic review principles were identified and are discussed. Overall, this work illustrates the advantages of applying structured and pre-defined methods for study evaluation and provides a unique case study comparing the impact of using different tools for evaluating animal toxicity studies.

[1]  Thomas Hartung,et al.  "ToxRTool", a new tool to assess the reliability of toxicological data. , 2009, Toxicology letters.

[2]  D. Weed.  Weight of Evidence: A Review of Concept and Methods , 2005, Risk analysis : an official publication of the Society for Risk Analysis.

[3]  Julie E Goodman,et al.  Systematic comparison of study quality criteria. , 2016, Regulatory toxicology and pharmacology : RTP.

[4]  T. Woodruff,et al.  Instruments for Assessing Risk of Bias and Other Methodological Criteria of Published Animal Studies: A Systematic Review , 2013, Environmental health perspectives.

[5]  U. Tillmann,et al.  A systematic approach for evaluating the quality of experimental toxicological and ecotoxicological data. , 1997, Regulatory toxicology and pharmacology : RTP.

[6]  A Beronius,et al.  Making the most of expert judgment in hazard and risk assessment of chemicals , 2017, Toxicology research.

[7]  Marlene Ågerstrand,et al.  Weight of evidence evaluation and systematic review in EU chemical risk assessment: Foundation is laid but guidance is needed. , 2016, Environment international.

[8]  Richard A. Becker,et al.  A survey of frameworks for best practices in weight-of-evidence analyses , 2013, Critical reviews in toxicology.

[9]  M. Page,et al.  Tools for assessing risk of reporting biases in studies and syntheses of studies: a systematic review , 2018, BMJ Open.

[10]  Matthias Greiner,et al.  Guidance on the use of the weight of evidence approach in scientific assessments , 2017, EFSA journal. European Food Safety Authority.

[11]  C. Gill,et al.  Systematic Differences between Cochrane and Non-Cochrane Meta-Analyses on the Same Topic: A Matched Pair Analysis , 2015, PloS one.

[12]  P. Tugwell,et al.  Non‐randomized studies as a source of complementary, sequential or replacement evidence for randomized controlled trials in systematic reviews on the effects of interventions , 2013, Research synthesis methods.

[13]  Tracey J. Woodruff,et al.  The Navigation Guide Systematic Review Methodology: A Rigorous and Transparent Method for Translating Environmental Health Science into Better Health Outcomes , 2014, Environmental health perspectives.

[14]  A R Jadad,et al.  Assessing the quality of randomized controlled trials: an annotated bibliography of scales and checklists. , 1995, Controlled clinical trials.

[15]  Marlene Ågerstrand,et al.  Study sensitivity: Evaluating the ability to detect effects in systematic reviews of chemical exposures. , 2016, Environment international.

[16]  David Moher,et al.  Reducing waste from incomplete or unusable reports of biomedical research , 2014, The Lancet.

[17]  Philip B. Stark,et al.  Before reproducibility must come preproducibility , 2018, Nature.

[18]  M. Rovers,et al.  SYRCLE’s risk of bias tool for animal studies , 2014, BMC Medical Research Methodology.

[19]  Erin E. Yost,et al.  Hazards of diisobutyl phthalate (DIBP) exposure: A systematic review of animal toxicology studies , 2018, Environment international.

[20]  S. Hopewell,et al.  Cochrane methods - twenty years experience in developing systematic review methods , 2013, Systematic Reviews.

[21]  Asbjørn Hróbjartsson,et al.  Lack of blinding of outcome assessors in animal model experiments implies risk of observer bias. , 2014, Journal of clinical epidemiology.

[22]  Marlene Ågerstrand,et al.  Implementing systematic review techniques in chemical risk assessment: Challenges, opportunities and recommendations. , 2016, Environment international.

[23]  Patrice Sutton,et al.  An evidence-based medicine methodology to bridge the gap between clinical and environmental health sciences. , 2011, Health affairs.

[24]  L. Haws,et al.  Role of Risk of Bias in Systematic Review for Chemical Risk Assessment: A Case Study in Understanding the Relationship Between Congenital Heart Defects and Exposures to Trichloroethylene , 2018, International journal of toxicology.

[25]  Holger J Schünemann,et al.  How credible are the study results? Evaluating and applying internal validity tools to literature-based assessments of environmental health hazards. , 2016, Environment international.

[26]  P. Griffiths,et al.  Ensuring the reporting quality of publications in nursing journals: A shared responsibility? , 2015, International journal of nursing studies.

[27]  Tasha R. Stanton,et al.  Scales to Assess the Quality of Randomized Controlled Trials: A Systematic Review , 2008, Physical Therapy.

[28]  Anna Beronius,et al.  Testing and refining the Science in Risk Assessment and Policy (SciRAP) web‐based platform for evaluating the reliability and relevance of in vivo toxicity studies , 2018, Journal of applied toxicology : JAT.

[29]  Sebastian Hoffmann,et al.  Guidance on assessing the methodological and reporting quality of toxicologically relevant studies: A scoping review. , 2016, Environment international.

[30]  Marlene Ågerstrand,et al.  Criteria for Reporting and Evaluating ecotoxicity Data (CRED): comparison and perception of the Klimisch and CRED methods for evaluating reliability and relevance of ecotoxicity studies , 2016, Environmental Sciences Europe.

[31]  Marlene Ågerstrand,et al.  A call for action: Improve reporting of research studies to increase the scientific basis for regulatory decision‐making , 2018, Journal of applied toxicology : JAT.

[32]  John R. Bucher,et al.  Systematic Review and Evidence Integration for Literature-Based Environmental Health Science Assessments , 2014, Environmental health perspectives.

[33]  Marlene Ågerstrand,et al.  Science in Risk Assessment and Policy (SciRAP): An Online Resource for Evaluating and Reporting In Vivo (Eco)Toxicity Studies , 2015 .