Minimising biases in expert elicitations to inform environmental management: Case studies from environmental flows in Australia

Abstract Environmental managers often do not have sufficient empirical data to inform decisions, and instead must rely on expert predictions. However, the informal methods often used to gather expert opinions are prone to cognitive and motivational biases. We developed a structured elicitation protocol, where opinions are directly incorporated into Bayesian Network (BBN) models. The 4-stage protocol includes approaches to minimise biases during pre-elicitation, workshop facilitation and output analysis; and results in a fully functional BBN model. We illustrate our protocol using examples from environmental flow management in Australia, presenting models of vegetation responses to changes in riverine flow regimes. The reliance on expert opinion and the contested nature of many environmental management decisions mean that our structured elicitation protocol is potentially of great value for developing robust environmental recommendations. This method also lends itself to effective adaptive management, because the expert-populated ecological response models can be readily updated with field data.

[1]  Mark A. Burgman,et al.  Redefining expertise and improving ecological judgment , 2011 .

[2]  Jeremy E. Oakley,et al.  Uncertain Judgements: Eliciting Experts' Probabilities , 2006 .

[3]  D. Kahneman,et al.  Measures of Anchoring in Estimation Tasks , 1995 .

[4]  Joshua Klayman,et al.  Overconfidence in interval estimates. , 2004, Journal of experimental psychology. Learning, memory, and cognition.

[5]  Erik Lebret,et al.  The use of expert elicitation in environmental health impact assessment: a seven step procedure , 2010, Environmental health : a global access science source.

[6]  Ankush Mittal,et al.  Bayesian Network Technologies: Applications and Graphical Models , 2007 .

[7]  Mark A. Burgman,et al.  Trusting Judgements: How to Get the Best out of Experts , 2015 .

[8]  R. Sparks,et al.  THE NATURAL FLOW REGIME. A PARADIGM FOR RIVER CONSERVATION AND RESTORATION , 1997 .

[9]  Fiona Fidler,et al.  Reducing Overconfidence in the Interval Judgments of Experts , 2010, Risk analysis : an official publication of the Society for Risk Analysis.

[10]  Ralph L. Keeney,et al.  Eliciting probabilities from experts in complex technical problems , 1991 .

[11]  David J. Hand,et al.  Statistics: A Very Short Introduction , 2008 .

[12]  N. Poff,et al.  Ecological responses to altered flow regimes: a literature review to inform the science and management of environmental flows , 2010 .

[13]  N. Dalkey,et al.  An Experimental Application of the Delphi Method to the Use of Experts , 1963 .

[14]  B. Marcot,et al.  Bayesian belief networks: applications in ecology and natural resource management , 2006 .

[15]  N. Poff,et al.  Squeezing the most out of existing literature: a systematic re‐analysis of published evidence on ecological responses to altered flows , 2013 .

[16]  K. Mengersen,et al.  Eliciting Expert Knowledge in Conservation Science , 2012, Conservation biology : the journal of the Society for Conservation Biology.

[17]  Jeremy E. Oakley,et al.  Probability is perfect, but we can't elicit it perfectly , 2004, Reliab. Eng. Syst. Saf..

[18]  Tara G Martin,et al.  A guide to eliciting and using expert knowledge in Bayesian ecological models. , 2010, Ecology letters.

[19]  Michael J. Stewardson,et al.  Informing Environmental Water Management Decisions: Using Conditional Probability Networks to Address the Information Needs of Planning and Implementation Cycles , 2018, Environmental Management.

[20]  Tim Bedford,et al.  Probabilistic Inversion of Expert Judgments in the Quantification of Model Uncertainty , 2005, Manag. Sci..

[21]  J. Langford,et al.  Legislating for sustainable basin management: the story of Australia's Water Act (2007) , 2013 .

[22]  R. Tharme A global perspective on environmental flow assessment: emerging trends in the development and application of environmental flow methodologies for rivers , 2003 .

[23]  Robin M. Hogarth,et al.  Cognitive Processes and the Assessment of Subjective Probability Distributions , 1975 .

[24]  A. Horne Water for the Environment : From Policy and Science to Implementation and Management , 2017 .

[25]  I. J. Myung,et al.  Maximum Entropy Aggregation of Expert Predictions , 1996 .

[26]  M. Kynn The ‘heuristics and biases’ bias in expert elicitation , 2007 .

[27]  A. Arthington,et al.  Flow restoration and protection in Australian rivers , 2003 .

[28]  M. Acreman,et al.  Environmental flows and the European Water Framework Directive. , 2010 .

[29]  M. Burgman Risks and Decisions for Conservation and Environmental Management: Experts, stakeholders and elicitation , 2005 .

[30]  Daryl L. Nielsen,et al.  Scoping the development of a method to assess the relative environmental benefits of Complementary Measures , 2017 .

[31]  Barry T. Hart,et al.  Bayesian network models for environmental flow decision‐making: 1. Latrobe River Australia , 2011 .

[32]  Andreas Mojzisch,et al.  Why groups perform better than individuals at quantitative judgment tasks: Group-to-individual transfer as an alternative to differential weighting , 2012 .

[33]  M. Kennard,et al.  Bayesian network models for environmental flow decision making in the Daly River, Northern Territory, Australia , 2012 .

[34]  Kevin B. Korb,et al.  Bayesian Artificial Intelligence , 2004, Computer science and data analysis series.

[35]  R. L. Winkler,et al.  Advances in Decision Analysis: Aggregating Probability Distributions , 2007 .

[36]  Laurence Smith,et al.  The role of expert opinion in environmental modelling , 2012, Environ. Model. Softw..

[37]  B. L. Lamb,et al.  Detecting biological responses to flow management: missed opportunities; future directions , 2008 .

[38]  N. Poff,et al.  A General Approach to Predicting Ecological Responses to Environmental Flows: Making Best Use of the Literature, Expert Knowledge, and Monitoring Data , 2015 .

[39]  Evan Harrison,et al.  Analyzing cause and effect in environmental assessments: using weighted evidence from the literature , 2011, Freshwater Science.

[40]  Kerrie Mengersen,et al.  Elicitator: A User-Friendly, Interactive Tool to Support Scenario-Based Elicitation of Expert Knowledge , 2012, HiPC 2012.

[41]  Joslin L. Moore,et al.  Improving visual estimation through active feedback , 2013 .

[42]  Mark A. Burgman,et al.  Evaluating the accuracy and calibration of expert predictions under uncertainty: predicting the outcomes of ecological research , 2012 .

[43]  J. Cain Planning improvements in natural resource management. Guidelines for using Bayesian networks to support the planning and management of development programmes in the water sector and beyond , 2001 .

[44]  Sarah J. Converse,et al.  Special Issue Article: Adaptive management for biodiversity conservation in an uncertain world Which uncertainty? Using expert elicitation and expected value of information to design an adaptive program , 2011 .

[45]  Mark A. Burgman,et al.  I nvestigate D iscuss E stimate A ggregate for structured expert judgement , 2017 .

[46]  J. Webb,et al.  Environmental Flows Can Reduce the Encroachment of Terrestrial Vegetation into River Channels: A Systematic Literature Review , 2013, Environmental Management.

[47]  Marc Hockings,et al.  Managers consider multiple lines of evidence important for biodiversity management decisions. , 2012, Journal of environmental management.

[48]  Yacov Salomon Unimodal density estimation with applications in expert elicitation and decision making under uncertainty , 2013 .

[49]  M. McCarthy Bayesian Methods for Ecology , 2007 .

[50]  J. Pearl Causality: Models, Reasoning and Inference , 2000 .

[51]  Barry T. Hart,et al.  A Bayesian network approach to support environmental flow restoration decisions in the Yarra River, Australia , 2013, Stochastic Environmental Research and Risk Assessment.

[52]  A. O'Hagan,et al.  Statistical Methods for Eliciting Probability Distributions , 2005 .

[53]  Kerrie L. Mengersen,et al.  Elicitator: An expert elicitation tool for regression in ecology , 2010, Environ. Model. Softw..

[54]  Alysson M. Costa,et al.  Using optimization to develop a "designer" environmental flow regime , 2017, Environ. Model. Softw..

[55]  Mark A. Burgman,et al.  What Is Expert Knowledge, How Is Such Knowledge Gathered, and How Do We Use It to Address Questions in Landscape Ecology? , 2012 .