The use of expert elicitation in environmental health impact assessment: a seven step procedure

BackgroundEnvironmental health impact assessments often have to deal with substantial uncertainties. Typically, the knowledge-base is limited with incomplete, or inconsistent evidence and missing or ambiguous data. Consulting experts can help to identify and address uncertainties.MethodsFormal expert elicitation is a structured approach to systematically consult experts on uncertain issues. It is most often used to quantify ranges for poorly known parameters, but may also be useful to further develop qualitative issues such as definitions, assumptions or conceptual (causal) models. A thorough preparation and systematic design and execution of an expert elicitation process may increase the validity of its outcomes and transparency and trustworthiness of its conclusions. Various expert elicitation protocols and methods exist. However, these are often not universally applicable, and need customization to suite the needs of a specific study. In this paper, we set out to develop a widely applicable method for the use of expert elicitation in environmental health impact assessment.ResultsWe present a practical yet flexible seven step procedure towards organising expert elicitation in the context of environmental health impact assessment, based on existing protocols. We describe how customization for specific applications is always necessary. In particular, three issues affect the choice of methods for a particular application: the types of uncertainties considered, the intended use of the elicited information, and the available resources. We outline how these three considerations guide choices regarding the design and execution of expert elicitation. We present signposts to sources where the issues are discussed in more depth to give the newcomer the insights needed to make the protocol work. The seven step procedure is illustrated using examples from earlier published elicitations in the field of environmental health research.ConclusionsWe conclude that, despite some known criticism on its validity, formal expert elicitation can support environmental health research in various ways. Its main purpose is to provide a temporary summary of the limited available knowledge, which can serve as a provisional basis for policy until further research has been carried out.

[1]  Henri M. Ami.  THE ROYAL SOCIETY OF CANADA. , 1893 .

[2]  Elizabeth A. Casman,et al.  Elicitation of Expert Judgments of Uncertainty in the Risk Assessment of Herbicide‐Tolerant Oilseed Crops , 2004, Risk analysis : an official publication of the Society for Risk Analysis.

[3]  Roy G Beran,et al.  The role of the expert witness in the adversarial legal system. , 2009, Journal of law and medicine.

[4]  Matthias Kaiser,et al.  Diagnosing and prioritizing uncertainties according to their relevance for policy: the case of transgene silencing. , 2008, The Science of the total environment.

[5]  Lisa Conner,et al.  Expert judgment assessment of the mortality impact of changes in ambient fine particulate matter in the U.S. , 2008, Environmental science & technology.

[6]  Erik Lebret,et al.  Dealing with uncertainties in environmental burden of disease assessment , 2009, Environmental health : a global access science source.

[7]  Roy C. Amara,et al.  Some views on the use of expert judgment , 1971 .

[8]  J. Ravetz,et al.  RIVM / MNP Guidance for Uncertainty Assessment and Communication : Tool Catalogue for Uncertainty Assessment , 2004 .

[9]  Martin Steinert,et al.  A dissensus based online Delphi approach: An explorative research tool , 2009 .

[10]  Ralph L. Keeney,et al.  Eliciting probabilities from experts in complex technical problems , 1991 .

[11]  Jack B. Soll,et al.  Overconfidence: It Depends on How, What, and Whom You Ask. , 1999, Organizational behavior and human decision processes.

[12]  Elissa Beerbohm.  A Pilot Expert Elicitation to Assess the Risks of Malaria Vector Control Strategies in East Africa , 2007 .

[13]  G. Katona,et al.  The Art of Asking Questions , 1951 .

[14]  J. Sluijs,et al.  The Inclusion of Stakeholder Knowledge and Perspectives in Integrated Assessment of Climate Change , 2006 .

[15]  Helen M. Regan,et al.  A TAXONOMY AND TREATMENT OF UNCERTAINTY FOR ECOLOGY AND CONSERVATION BIOLOGY , 2002 .

[16]  Funtowicz Silvio.  Post-Normal Science , 2020, Encyclopedia of Creativity, Invention, Innovation and Entrepreneurship.

[17]  Khara D Grieger,et al.  Conscious worst case definition for risk assessment, part I: a knowledge mapping approach for defining most critical risk factors in integrative risk management of chemicals and nanomaterials. , 2010, The Science of the total environment.

[18]  K. Vind.  A foundation for statistics , 2003 .

[19]  Enrico Zio,et al.  Two methods for the structured assessment of model uncertainty by experts in performance assessments of radioactive waste repositories , 1996 .

[20]  Carl-Axel S. Staël von Holstein,et al.  Exceptional Paper---Probability Encoding in Decision Analysis , 1975 .

[21]  Kaplan,et al.  ‘Combining Probability Distributions from Experts in Risk Analysis’ , 2000, Risk analysis : an official publication of the Society for Risk Analysis.

[22]  Simon Peck,et al.  Group Model Building: Facilitating Team Learning Using System Dynamics , 1996, J. Oper. Res. Soc..

[23]  Kerrie Mengersen,et al.  Geographically Assisted Elicitation of Expert Opinion for Regression Models , 2007 .

[24]  S. Moore,et al.  Nuclear Regulatory Commission , 1975 .

[25]  Roger M Cooke,et al.  Attribution of foodborne pathogens using structured expert elicitation. , 2008, Foodborne pathogens and disease.

[26]  A. Tversky,et al.  The framing of decisions and the psychology of choice. , 1981, Science.

[27]  D. Cook,et al.  Effect of the framing of questionnaire items regarding satisfaction with training on residents' responses. , 1999, Academic medicine : journal of the Association of American Medical Colleges.

[28]  D. Briggs,et al.  A framework for integrated environmental health impact assessment of systemic risks , 2008, Environmental health : a global access science source.

[29]  Roger M. Cooke,et al.  Probabilistic inversion for chicken processing lines , 2006, Reliab. Eng. Syst. Saf..

[30]  Kerrie L. Mengersen,et al.  Elicitator: An expert elicitation tool for regression in ecology , 2010, Environ. Model. Softw..

[31]  J. Evans,et al.  Use of expert judgment in exposure assessment. Part I. Characterization of personal exposure to benzene , 2001, Journal of Exposure Analysis and Environmental Epidemiology.

[32]  J. A. Wardekker,et al.  Uncertainty Communication : Issues and good practice , 2007 .

[33]  F R Jelovsek,et al.  Eliciting principles of hazard identification from experts. , 1990, Teratology.

[34]  David W. Keith,et al.  When is it appropriate to combine expert judgments? , 1996 .

[35]  A. Tversky,et al.  Judgment under Uncertainty: Heuristics and Biases , 1974, Science.

[36]  Ronald E. Doel,et al.  :The Honest Broker: Making Sense of Science in Policy and Politics , 2008 .

[37]  J S Evans,et al.  Use of probabilistic expert judgment in uncertainty analysis of carcinogenic potency. , 1994, Regulatory toxicology and pharmacology : RTP.

[38]  Tze-Yun Leong,et al.  PROBES: a framework for probability elicitation from experts , 1999, AMIA.

[39]  Janssen Phm,et al.  Uncertainty analysis for NOx emissions from Dutch passenger cars in 1998. Applying structured expert elicitation and distinguishing different types of uncertainty , 2004 .

[40]  Warren E. Walker,et al.  Defining Uncertainty: A Conceptual Basis for Uncertainty Management in Model-Based Decision Support , 2003 .

[41]  Ronald L. Iman,et al.  Expert opinion in risk analysis: the NUREG-1150 methodology , 1989 .

[42]  J R Ravetz,et al.  A guidance for assessing and communicating uncertainties. , 2005, Water science and technology : a journal of the International Association on Water Pollution Research.

[43]  W.Owen Nutt.  Business planning for an uncertain future , 1984 .

[44]  George Wright,et al.  The Delphi technique as a forecasting tool: issues and analysis , 1999 .

[45]  Erik Lebret,et al.  Expert elicitation on ultrafine particles: likelihood of health effects and causal pathways , 2009, Particle and Fibre Toxicology.

[46]  Arthur C. Petersen,et al.  A method for the analysis of assumptions in model-based environmental assessments , 2011, Environ. Model. Softw..

[47]  J. Crisp,et al.  The Delphi method? , 1997, Nursing research.

[48]  S. Presser,et al.  Questions and Answers in Attitude Surveys: Experiments on Question Form, Wording, and Context , 1996 .

[49]  Thomas Oberlechner.  Psychology of Judgment and Decision-Making , 2006 .

[50]  Kerrie Mengersen,et al.  Comparison of three expert elicitation methods for logistic regression on predicting the presence of the threatened brush‐tailed rock‐wallaby Petrogale penicillata , 2009 .

[51]  J D Graham,et al.  Expert scientific judgment and cancer risk assessment: a pilot study of pharmacokinetic data. , 1988, Risk analysis : an official publication of the Society for Risk Analysis.

[52]  Sandra Hoffmann,et al.  Using expert elicitation to link foodborne illnesses in the United States to foods. , 2007, Journal of food protection.

[53]  Jerome R. Ravetz,et al.  Exploring the quality of evidence for complex and contested policy decisions , 2008 .

[54]  Victoria Wibeck,et al.  Communicating Uncertainty: Models of Communication and the Role of Science in Assessing Progress towards Environmental Objectives , 2009 .

[55]  Milind Kandlikar,et al.  Expressions of likelihood and confidence in the IPCC uncertainty assessment process , 2007 .

[56]  M. Schmid,et al.  Good Practice Guidance and Uncertainty Management in National Greenhouse Gas Inventories Indirect N 2 O Emissions from Agriculture 381 INDIRECT N 2 O EMISSIONS FROM AGRICULTURE , 2001 .

[57]  Bert Brunekreef,et al.  Concentration response functions for ultrafine particles and all-cause mortality and hospital admissions: results of a European expert panel elicitation. , 2009, Environmental science & technology.

[58]  Thomas Gilovich,et al.  Peering Into the Bias Blind Spot: People’s Assessments of Bias in Themselves and Others , 2005, Personality & social psychology bulletin.

[59]  Gene Rowe,et al.  Perspectives on Expertise in the Aggregation of Judgments , 1992 .

[60]  1986 Part II Environmental Protection Agency Guidelines for Carcinogen Risk Assessment , 2022 .

[61]  H. J. van der Fels-Klerx,et al.  A Structured Expert Judgment Study for a Model of Campylobacter Transmission During Broiler‐Chicken Processing , 2005, Risk analysis : an official publication of the Society for Risk Analysis.

[62]  S. Funtowicz,et al.  Combining Quantitative and Qualitative Measures of Uncertainty in Model‐Based Environmental Assessment: The NUSAP System , 2005, Risk analysis : an official publication of the Society for Risk Analysis.

[63]  Elizabeth C. Hirschman,et al.  Judgment under Uncertainty: Heuristics and Biases , 1974, Science.

[64]  P. Powell.  Expertise and Decision Support , 2013 .

[65]  Allan Mazur,et al.  II Disputes between experts , 1973 .

[66]  B. Brown,et al.  THE DELPHI METHOD, II: STRUCTURE OF EXPERIMENTS, , 1969 .

[67]  J. R. Ravets,et al.  Post-Normal Science , 2006 .

[68]  Y A R O M A N,et al.  Expert Judgment Assessment of the Mortality Impact of Changes in Ambient Fine Particulate Matter in the , 2008 .

[69]  Norman Crolee Dalkey,et al.  An experimental study of group opinion , 1969 .

[70]  G. Annas,et al.  Judging Science: Scientific Knowledge and the Federal Courts , 1999, Nature Medicine.

[71]  Max Henrion,et al.  Uncertainty: A Guide to Dealing with Uncertainty in Quantitative Risk and Policy Analysis , 1990 .

[72]  C. Vionnet.  [Environmental health]. , 1983, Krankenpflege. Soins infirmiers.

[73]  L. J. Savage,et al.  The Foundation of Statistics , 1956 .

[74]  Clive E Sabel,et al.  Uncertainty in epidemiology and health risk and impact assessment , 2009, Environmental geochemistry and health.

[75]  J. Barendregt,et al.  Global burden of disease , 1997, The Lancet.

[76]  Peter A. Vanrolleghem,et al.  Uncertainty in the environmental modelling process - A framework and guidance , 2007, Environ. Model. Softw..

[77]  Anne B. Knol,et al.  Expert elicitation: methodological suggestions for its use in environmental health impact assessments , 2008 .

[78]  Roger M. Cooke,et al.  On the performance of social network and likelihood-based expert weighting schemes , 2008, Reliab. Eng. Syst. Saf..

[79]  Irwin Streiff,et al.  Environmental Health service , 1974 .

[80]  Grant T Baldwin,et al.  Understanding Wicked Problems: A Key to Advancing Environmental Health Promotion , 2004, Health education & behavior : the official publication of the Society for Public Health Education.

[81]  GERD,et al.  New Directions in Judgment and Decision Making The Psychology of Good Judgment : Frequency Formats and Simple Algorithms , 2006 .

[82]  James G. Titus,et al.  The risk of sea level rise , 1996 .

[83]  William C. Clark,et al.  Public Participation in Sustainability Science: Frontmatter , 2003 .

[84]  S. Hora.  Advances in Decision Analysis: Eliciting Probabilities from Experts , 2007 .

[85]  A. Petersen.  Simulating Nature: A Philosophical Study of Computer-Simulation Uncertainties and Their Role in Climate Science and Policy Advice, Second Edition , 2007 .

[86]  Karen Cortell Reisman.  "The art of asking questions". , 2008, Texas dental journal.

[87]  A. Spence,et al.  Implicit attitudes towards genetically modified (GM) foods: A comparison of context-free and context-dependent evaluations , 2006, Appetite.

[88]  Marko Tainio,et al.  A probabilistic characterization of the relationship between fine particulate matter and mortality: elicitation of European experts. , 2007, Environmental science & technology.

[89]  M. P. Lee,et al.  Branch technical position on the use of expert elicitation in the high-level radioactive waste program , 1996 .

[90]  J. A. Wardekker,et al.  Uncertainty communication in environmental assessments: views from the Dutch science-policy interface , 2008 .

[91]  R. Cooke.  Experts in Uncertainty: Opinion and Subjective Probability in Science , 1991 .

[92]  J. Charles Kerkering,et al.  Eliciting and Analyzing Expert Judgment, A Practical Guide , 2002, Technometrics.