Draft for internal testing Scientific Committee guidance on appraising and integrating evidence from epidemiological studies for use in EFSA's scientific assessments

Abstract EFSA requested its Scientific Committee to prepare a guidance document on appraising and integrating evidence from epidemiological studies for use in EFSA's scientific assessments. The guidance document provides an introduction to epidemiological studies and illustrates the typical biases of the different epidemiological study designs. It describes key epidemiological concepts relevant for evidence appraisal. Regarding study reliability, measures of association, exposure assessment, statistical inferences, systematic error and effect modification are explained. Regarding study relevance, the guidance describes the concept of external validity. The principles of appraising epidemiological studies are illustrated, and an overview of Risk of Bias (RoB) tools is given. A decision tree is developed to assist in the selection of the appropriate Risk of Bias tool, depending on study question, population and design. The customisation of the study appraisal process is explained, detailing the use of RoB tools and assessing the risk of bias in the body of evidence. Several examples of appraising experimental and observational studies using a Risk of Bias tool are annexed to the document to illustrate the application of the approach. This document constitutes a draft that will be applied in EFSA's assessments during a 1‐year pilot phase and be revised and complemented as necessary. Before finalisation of the document, a public consultation will be launched.

[1]  Scott Henderson,et al.  Pathology, Causes, Course and Treatment of Rheumatism , 1873, The Chicago medical journal.

[2]  C. Bragard,et al.  Update of the Xylella spp. host plant database – systematic literature search up to 30 June 2019 , 2020, EFSA journal. European Food Safety Authority.

[3]  P. Medina-Pastor,et al.  The 2018 European Union report on pesticide residues in food , 2020, EFSA journal. European Food Safety Authority.

[4]  K. Nagy,et al.  The European Union Summary Report on Antimicrobial Resistance in zoonotic and indicator bacteria from humans, animals and food in 2017/2018 , 2020, EFSA journal. European Food Safety Authority.

[5]  G. Schrader,et al.  Pest survey card on Agrilus anxius , 2020 .

[6]  M. Arancibia,et al.  General concepts in biostatistics and clinical epidemiology: observational studies with cohort design. , 2019, Medwave.

[7]  M. Greiner,et al.  Refining animal research: The Animal Study Registry , 2019, PLoS biology.

[8]  Spiros C. Denaxas,et al.  Avoidable flaws in observational analyses: an application to statins and cancer , 2019, Nature Medicine.

[9]  G. Guyatt,et al.  Effect of Lower Versus Higher Red Meat Intake on Cardiometabolic and Cancer Outcomes , 2019, Annals of Internal Medicine.

[10]  T. Trikalinos,et al.  The Problem with Mechanistic Risk of Bias Assessments in Evidence Synthesis of Observational Studies and a Practical Alternative: Assess the Impact of Specific Sources of Potential Bias. , 2019, American journal of epidemiology.

[11]  D. Spiegelman,et al.  One-stage dose–response meta-analysis for aggregated data , 2019, Statistical methods in medical research.

[12]  S. Parnell,et al.  Guidance on commodity risk assessment for the evaluation of high risk plants dossiers , 2019, EFSA journal. European Food Safety Authority.

[13]  Sander Greenland,et al.  Scientists rise up against statistical significance , 2019, Nature.

[14]  D. Scott,et al.  Critical appraisal of nonrandomized studies—A review of recommended and commonly used tools , 2019, Journal of evaluation in clinical practice.

[15]  A. Hansell,et al.  A risk of bias instrument for non-randomized studies of exposures: A users’ guide to its application in the context of GRADE , 2018, Environment international.

[16]  MD Bryan Kestenbaum Epidemiology and Biostatistics , 2019, Springer International Publishing.

[17]  K. Nagy,et al.  The European Union summary report on trends and sources of zoonoses, zoonotic agents and food‐borne outbreaks in 2017 , 2018, EFSA journal. European Food Safety Authority.

[18]  T. Woodruff,et al.  The risk of bias in observational studies of exposures (ROBINS-E) tool: concerns arising from application to observational studies of exposures , 2018, Systematic Reviews.

[19]  Heather M. Wallace,et al.  Risk for animal and human health related to the presence of dioxins and dioxin‐like PCBs in feed and food , 2018, EFSA journal. European Food Safety Authority.

[20]  D. Savitz,et al.  Serum Perfluorooctanoic Acid and Birthweight: An Updated Meta-analysis With Bias Analysis , 2018, Epidemiology.

[21]  A. Berlin,et al.  Potato bacterial wilt in Rwanda: occurrence, risk factors, farmers’ knowledge and attitudes , 2018, Food Security.

[22]  Mélanie Frappier,et al.  The Book of Why: The New Science of Cause and Effect , 2018, Science.

[23]  Holger J. Schünemann,et al.  Identifying the PECO: A framework for formulating good questions to explore the association of environmental and other exposures with health outcomes. , 2018, Environment international.

[24]  S. Savva,et al.  National dietary survey on the adult population of Cyprus , 2018, EFSA Supporting Publications.

[25]  J. West,et al.  Guidance on quantitative pest risk assessment , 2018, EFSA journal. European Food Safety Authority.

[26]  Antoine Messéan,et al.  Implementation of PROMETHEUS 4‐step approach for evidence use in EFSA scientific assessments: benefits, issues, needs and solutions , 2018 .

[27]  M. Page,et al.  Tools for assessing risk of reporting biases in studies and syntheses of studies: a systematic review , 2018, BMJ Open.

[28]  Melanie Gross,et al.  Weight of evidence approaches for the identification of endocrine disrupting properties of chemicals: Review and recommendations for EU regulatory application , 2017, Regulatory toxicology and pharmacology : RTP.

[29]  M. Weisskopf,et al.  Trade-offs of Personal Versus More Proxy Exposure Measures in Environmental Epidemiology , 2017, Epidemiology.

[30]  S. More,et al.  Guidance on the assessment of the biological relevance of data in scientific assessments , 2017, EFSA journal. European Food Safety Authority.

[31]  Matthias Greiner,et al.  Guidance on the use of the weight of evidence approach in scientific assessments , 2017, EFSA journal. European Food Safety Authority.

[32]  Y. Li,et al.  Half-lives of PFOS, PFHxS and PFOA after end of exposure to contaminated drinking water , 2017, Occupational and Environmental Medicine.

[33]  K. Jager,et al.  Relative risk versus absolute risk: one cannot be interpreted without the other: Clinical Epidemiology in Nephrology , 2017, Nephrology, dialysis, transplantation : official publication of the European Dialysis and Transplant Association - European Renal Association.

[34]  José Cortiñas Abrahantes,et al.  Update: use of the benchmark dose approach in risk assessment , 2017, EFSA journal. European Food Safety Authority.

[35]  Julian P T Higgins,et al.  Biases in Randomized Trials: A Conversation Between Trialists and Epidemiologists. , 2017, Epidemiology.

[36]  M. Hernán,et al.  ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions , 2016, British Medical Journal.

[37]  N. Orsini,et al.  Dose-response meta-analysis of differences in means , 2016, BMC Medical Research Methodology.

[38]  Sebastian Hoffmann,et al.  Guidance on assessing the methodological and reporting quality of toxicologically relevant studies: A scoping review. , 2016, Environment international.

[39]  Nancy Santesso,et al.  GRADE: Assessing the quality of evidence in environmental and occupational health. , 2016, Environment international.

[40]  S. Goodman,et al.  Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations , 2016, European Journal of Epidemiology.

[41]  K. Rothman,et al.  Disengaging from statistical significance , 2016, European Journal of Epidemiology.

[42]  Julie E Goodman,et al.  Systematic comparison of study quality criteria. , 2016, Regulatory toxicology and pharmacology : RTP.

[43]  Neil Pearce,et al.  Analysis of matched case-control studies , 2016, British Medical Journal.

[44]  Alex Broadbent,et al.  Causality and causal inference in epidemiology: the need for a pluralistic approach , 2016, International journal of epidemiology.

[45]  L. Hartley,et al.  Dietary fibre for the primary prevention of cardiovascular disease. , 2016, The Cochrane database of systematic reviews.

[46]  D. Bowen,et al.  HANDBOOK OF HEALTH DECISION SCIENCE , 2016 .

[47]  Jane Wardle,et al.  Effect of a behavioural intervention in obese pregnant women (the UPBEAT study): a multicentre, randomised controlled trial. , 2015, The lancet. Diabetes & endocrinology.

[48]  Melvin E. Andersen,et al.  Associations of Perfluoroalkyl Substances (PFAS) with Lower Birth Weight: An Evaluation of Potential Confounding by Glomerular Filtration Rate Using a Physiologically Based Pharmacokinetic Model (PBPK) , 2015, Environmental health perspectives.

[49]  B. Nørrung,et al.  Editorial: Increasing robustness, transparency and openness of scientific assessments , 2015 .

[50]  E. Boa,et al.  Better together: identifying the benefits of a closer integration between plant health, agriculture and One Health. , 2015 .

[51]  Efsa Publication,et al.  EFSA (European Food Safety Authority) and ECDC (European Centre for Disease Prevention and Control), 2015. The European Union summary report on trends and sources of zoonoses, zoonotic agents and food-borne outbreaks in 2014 , 2015 .

[52]  Tracey J. Woodruff,et al.  The Navigation Guide—Evidence-Based Medicine Meets Environmental Health: Systematic Review of Human Evidence for PFOA Effects on Fetal Growth , 2014, Environmental health perspectives.

[53]  Paula I. Johnson,et al.  The Navigation Guide—Evidence-Based Medicine Meets Environmental Health: Integration of Animal and Human Evidence for PFOA Effects on Fetal Growth , 2014, Environmental health perspectives.

[54]  Tracey J. Woodruff,et al.  The Navigation Guide Systematic Review Methodology: A Rigorous and Transparent Method for Translating Environmental Health Science into Better Health Outcomes , 2014, Environmental health perspectives.

[55]  Paula I. Johnson,et al.  The Navigation Guide—Evidence-Based Medicine Meets Environmental Health: Systematic Review of Nonhuman Evidence for PFOA Effects on Fetal Growth , 2014, Environmental health perspectives.

[56]  John R. Bucher,et al.  Systematic Review and Evidence Integration for Literature-Based Environmental Health Science Assessments , 2014, Environmental health perspectives.

[57]  M. Rovers,et al.  SYRCLE’s risk of bias tool for animal studies , 2014, BMC Medical Research Methodology.

[58]  Ian M Thompson,et al.  Baseline selenium status and effects of selenium and vitamin e supplementation on prostate cancer risk. , 2014, Journal of the National Cancer Institute.

[59]  R Jeffrey Lewis,et al.  Mode of action human relevance (species concordance) framework: Evolution of the Bradford Hill considerations and comparative analysis of weight of evidence , 2014, Journal of applied toxicology : JAT.

[60]  J. Sommar,et al.  Hip Fracture Risk and Cadmium in Erythrocytes: A Nested Case–Control Study with Prospectively Collected Samples , 2013, Calcified Tissue International.

[61]  L. Hartling,et al.  Assessing Risk of Bias and Confounding in Observational Studies of Interventions or Exposures: Further Development of the RTI Item Bank , 2013 .

[62]  Peter J Boogaard,et al.  A systematic approach for evaluating and scoring human data. , 2013, Regulatory toxicology and pharmacology : RTP.

[63]  Christiane,et al.  World Medical Association Declaration of Helsinki: ethical principles for medical research involving human subjects. , 2013, JAMA.

[64]  M. Wolfe Informational Meeting on the Draft OHAT Approach for Systematic Review and Evidence Integration for Literature-based Health Assessments - February 2013 , 2013 .

[65]  Ian R White,et al.  Nonadherence to treatment protocol in published randomised controlled trials: a review , 2012, Trials.

[66]  D. Savitz,et al.  Relationship of Perfluorooctanoic Acid Exposure to Pregnancy Outcome Based on Birth Records in the Mid-Ohio Valley , 2012, Environmental health perspectives.

[67]  J. Clemente,et al.  The Impact of the Gut Microbiota on Human Health: An Integrative View , 2012, Cell.

[68]  A. Vickers,et al.  Against quantiles: categorization of continuous variables in epidemiologic research, and its discontents , 2012, BMC Medical Research Methodology.

[69]  S. Snedeker,et al.  Do Interactions Between Gut Ecology and Environmental Chemicals Contribute to Obesity and Diabetes? , 2011, Environmental health perspectives.

[70]  Nicola Orsini,et al.  Meta-analysis for linear and nonlinear dose-response relations: examples, an evaluation of approximations, and software. , 2012, American journal of epidemiology.

[71]  I. White,et al.  Departure from treatment protocol in published randomised controlled trials: a review , 2011, Trials.

[72]  V. Pīrāgs,et al.  Acute and multiple-dose studies to determine the safety, tolerability, and pharmacokinetic profile of advantame in healthy volunteers. , 2011, Food and chemical toxicology : an international journal published for the British Industrial Biological Research Association.

[73]  A. Lang,et al.  Caffeine in Parkinson's disease: A pilot open‐label, dose‐escalation study , 2011, Movement disorders : official journal of the Movement Disorder Society.

[74]  K. Steenland,et al.  Retrospective Exposure Estimation and Predicted versus Observed Serum Perfluorooctanoic Acid Concentrations for Participants in the C8 Health Project , 2011, Environmental health perspectives.

[75]  Alex Burdorf,et al.  A Meta-analysis of Asbestos and Lung Cancer: Is Better Quality Exposure Assessment Associated with Steeper Slopes of the Exposure–Response Relationships? , 2011, Environmental health perspectives.

[76]  N. Toft,et al.  Structured approach to design of diagnostic test evaluation studies for chronic progressive infections in animals. , 2011, Veterinary microbiology.

[77]  James A. Swenberg,et al.  Toxicology and Epidemiology: Improving the Science with a Framework for Combining Toxicological and Epidemiological Evidence to Establish Causal Inference , 2011, Toxicological sciences : an official journal of the Society of Toxicology.

[78]  N. Rothman,et al.  Occupational Benzene Exposure and the Risk of Lymphoma Subtypes: A Meta-analysis of Cohort Studies Incorporating Three Study Quality Dimensions , 2010, Environmental health perspectives.

[79]  Assessing the Risk of Bias of Individual Studies when Comparing Medical Interventions , 2011 .

[80]  J. Higgins Cochrane handbook for systematic reviews of interventions. Version 5.1.0 [updated March 2011]. The Cochrane Collaboration , 2011 .

[81]  M. Dhont History of oral contraception , 2010, The European journal of contraception & reproductive health care : the official journal of the European Society of Contraception.

[82]  R. Kane,et al.  A systematic review of tools used to assess the quality of observational studies that examine incidence or prevalence and risk factors for diseases. , 2010, Journal of clinical epidemiology.

[83]  I. Cuthill,et al.  Reporting : The ARRIVE Guidelines for Reporting Animal Research , 2010 .

[84]  D. Lanska Epidemiology and Biostatistics: An Introduction to Clinical Research , 2010 .

[85]  Gábor L. Lövei,et al.  Application of Systematic Review Methodology to Food and Feed Safety Assessments to Support Decision Making , 2010 .

[86]  J. Pearl Causal inference in statistics: An overview , 2009 .

[87]  Ian T Cousins,et al.  Comparative assessment of the global fate and transport pathways of long-chain perfluorocarboxylic acids (PFCAs) and perfluorocarboxylates (PFCs) emitted from direct sources. , 2009, Environmental science & technology.

[88]  J. Crowley,et al.  Effect of selenium and vitamin E on risk of prostate cancer and other cancers: the Selenium and Vitamin E Cancer Prevention Trial (SELECT). , 2009, JAMA.

[89]  B. Kestenbaum Comprar Epidemiology and Biostatistics · An Introduction to Clinical Research | Kestenbaum, Bryan | 9780387884325 | Springer , 2009 .

[90]  J. Mclaughlin,et al.  Dietary predictors of perfluorinated chemicals: a study from the Danish National Birth Cohort. , 2008, Environmental science & technology.

[91]  T. Rassaf,et al.  Sustained benefits in vascular function through flavanol-containing cocoa in medicated diabetic patients a double-masked, randomized, controlled trial. , 2008, Journal of the American College of Cardiology.

[92]  Eric Langlois,et al.  An interlaboratory study of perfluorinated alkyl compound levels in human plasma. , 2008, Environmental research.

[93]  A. Steckler,et al.  The importance of external validity. , 2008, American journal of public health.

[94]  D. Savitz Guest Editorial: Biomarkers of Perfluorinated Chemicals and Birth Weight , 2007, Environmental health perspectives.

[95]  Douglas G Altman,et al.  The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: guidelines for reporting observational studies. , 2014, International journal of surgery.

[96]  Jørn Olsen,et al.  Perfluorinated Chemicals and Fetal Growth: A Study within the Danish National Birth Cohort , 2007, Environmental health perspectives.

[97]  M. Pencina,et al.  Clinical utility of different lipid measures for prediction of coronary heart disease in men and women. , 2007, JAMA.

[98]  Rolf U. Halden,et al.  Cord Serum Concentrations of Perfluorooctane Sulfonate (PFOS) and Perfluorooctanoate (PFOA) in Relation to Weight and Size at Birth , 2007, Environmental health perspectives.

[99]  J. Higgins,et al.  Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography. , 2007, International journal of epidemiology.

[100]  A. Genaidy,et al.  An epidemiological appraisal instrument – a tool for evaluation of epidemiological studies , 2007, Ergonomics.

[101]  David Moher,et al.  Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews , 2007, BMC medical research methodology.

[102]  Laurence V. Madden,et al.  The study of plant disease epidemics , 2007 .

[103]  David Kriebel,et al.  Bias in occupational epidemiology studies , 2006, Occupational and Environmental Medicine.

[104]  C. Rochester,et al.  LOW-FAT DIETARY PATTERN AND RISK OF CARDIOVASCULAR DISEASE-THE WOMEN'S HEALTH INITIATIVE RANDOMIZED CONTROLLED DIETARY MODIFICATION TRIAL , 2006 .

[105]  J. Manson,et al.  Low-fat dietary pattern and risk of cardiovascular disease: the Women's Health Initiative Randomized Controlled Dietary Modification Trial. , 2006, JAMA.

[106]  J. Ioannidis Why Most Published Research Findings Are False , 2005, PLoS medicine.

[107]  Michael M Lieber,et al.  Designing the Selenium and Vitamin E Cancer Prevention Trial (SELECT). , 2005, Journal of the National Cancer Institute.

[108]  J. Robins,et al.  A Structural Approach to Selection Bias , 2004, Epidemiology.

[109]  Debra T. Silverman,et al.  Selection of controls in case-control studies. II. Types of controls. , 1992, American journal of epidemiology.

[110]  P. Bossuyt,et al.  BMC Medical Research Methodology , 2002 .

[111]  F. Song,et al.  Evaluating non-randomised intervention studies. , 2003, Health technology assessment.

[112]  B. Turnbull,et al.  Selenium supplementation, baseline plasma selenium status and incidence of prostate cancer: an analysis of the complete treatment period of the Nutritional Prevention of Cancer Trial , 2003, BJU international.

[113]  J. Hsia,et al.  Effects of hormone replacement therapy and antioxidant vitamin supplements on coronary atherosclerosis in postmenopausal women: A randomized controlled trial , 2002 .

[114]  Wolfgang Völkel,et al.  Metabolism and kinetics of bisphenol a in humans at low doses following oral administration. , 2002, Chemical research in toxicology.

[115]  N McKoy,et al.  Systems to rate the strength of scientific evidence. , 2002, Evidence report/technology assessment.

[116]  G. Sigurdsson,et al.  Liquorice-induced rise in blood pressure: a linear dose-response relationship , 2001, Journal of Human Hypertension.

[117]  Matthew J.W. Cock,et al.  Invasive alien species: a toolkit of best prevention and management practices , 2001 .

[118]  O. Diekmann,et al.  Mathematical Epidemiology of Infectious Diseases: Model Building, Analysis and Interpretation , 2000 .

[119]  M. Egger,et al.  The hazards of scoring the quality of clinical trials for meta-analysis. , 1999, JAMA.

[120]  T. Lancaster,et al.  The new NHS: Medical students say no , 1998 .

[121]  U. Tillmann,et al.  A systematic approach for evaluating the quality of experimental toxicological and ecotoxicological data. , 1997, Regulatory toxicology and pharmacology : RTP.

[122]  R. J. Hayes,et al.  Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. , 1995, JAMA.

[123]  D. Sackett,et al.  Cochrane Collaboration , 1994, BMJ.

[124]  D. Albanes,et al.  The effect of vitamin E and beta carotene on the incidence of lung cancer and other cancers in male smokers. , 1994, The New England journal of medicine.

[125]  J K McLaughlin,et al.  Selection of controls in case-control studies. II. Types of controls. , 1992, American journal of epidemiology.

[126]  J K McLaughlin,et al.  Selection of controls in case-control studies. I. Principles. , 1992, American journal of epidemiology.

[127]  J K McLaughlin,et al.  Selection of controls in case-control studies. III. Design options. , 1992, American journal of epidemiology.

[128]  P. Crosignani Effects of hormone replacement therapy , 1992, International journal of fertility.

[129]  J D Emerson,et al.  An empirical study of the possible relation of treatment differences to quality scores in controlled randomized clinical trials. , 1990, Controlled clinical trials.

[130]  Sander Greenland,et al.  Modern Epidemiology 3rd edition , 1986 .

[131]  A H Smith,et al.  Non-Hodgkin's lymphoma and exposure to phenoxyherbicides, chlorophenols, fencing work, and meat works employment: a case-control study. , 1986, British journal of industrial medicine.

[132]  M. Gardner,et al.  Statistical guidelines for contributors to medical journals. , 1983, British medical journal.

[133]  A. Lilienfeld Foundations of Epidemiology , 1980 .

[134]  A. Lilienfeld,et al.  Foundations of epidemiology. 2nd edition. , 1980 .

[135]  A. B. Hill The Environment and Disease: Association or Causation? , 1965, Proceedings of the Royal Society of Medicine.