Group elicitations yield more consistent, yet more uncertain experts in understanding risks to ecosystem services in New Zealand bays

The elicitation of expert judgment is an important tool for assessment of risks and impacts in environmental management contexts, and especially important as decision-makers face novel challenges where prior empirical research is lacking or insufficient. Evidence-driven elicitation approaches typically involve techniques to derive more accurate probability distributions under fairly specific contexts. Experts are, however, prone to overconfidence in their judgements. Group elicitations with diverse experts can reduce expert overconfidence by allowing cross-examination and reassessment of prior judgements, but groups are also prone to uncritical “groupthink” errors. When the problem context is underspecified the probability that experts commit groupthink errors may increase. This study addresses how structured workshops affect expert variability among and certainty within responses in a New Zealand case study. We find that experts’ risk estimates before and after a workshop differ, and that group elicitations provided greater consistency of estimates, yet also greater uncertainty among experts, when addressing prominent impacts to four different ecosystem services in coastal New Zealand. After group workshops, experts provided more consistent ranking of risks and more consistent best estimates of impact through increased clarity in terminology and dampening of extreme positions, yet probability distributions for impacts widened. The results from this case study suggest that group elicitations have favorable consequences for the quality and uncertainty of risk judgments within and across experts, making group elicitation techniques invaluable tools in contexts of limited data.

[1]  J. Hahn.  Victims Of Groupthink A Psychological Study Of Foreign Policy Decisions And Fiascoes , 2016 .

[2]  W. Sutherland,et al.  Policy advice: Use experts wisely , 2015, Nature.

[3]  Igor Linkov,et al.  Scientific convergence: dealing with the elephant in the room. , 2014, Environmental science & technology.

[4]  R. Pressey,et al.  Assessing interactions of multiple stressors when data are limited: A Bayesian belief network applied to coral reefs , 2014 .

[5]  Christophe Biernacki,et al.  Model-based clustering for multivariate partial ranking data , 2014 .

[6]  Christophe Biernacki,et al.  Rankcluster: An R Package for Clustering Multivariate Partial Rankings , 2014, R J..

[7]  M. G. Morgan.  Use (and abuse) of expert elicitation in support of decision making for public policy , 2014, Proceedings of the National Academy of Sciences.

[8]  Baruch Fischhoff,et al.  The science and practice of risk ranking , 2013 .

[9]  Julien Jacques,et al.  A generative model for rank data based on insertion sort algorithm , 2013, Comput. Stat. Data Anal..

[10]  F. Chapin,et al.  Design principles for social‐ecological transformation toward sustainability: lessons from New Zealand sense of place , 2012 .

[11]  K. Mengersen,et al.  Eliciting Expert Knowledge in Conservation Science , 2012, Conservation biology : the journal of the Society for Conservation Biology.

[12]  Mark A. Burgman,et al.  What Is Expert Knowledge, How Is Such Knowledge Gathered, and How Do We Use It to Address Questions in Landscape Ecology? , 2012 .

[13]  Mark A. Burgman,et al.  Expert Status and Performance , 2011, PloS one.

[14]  Mark A. Burgman,et al.  Redefining expertise and improving ecological judgment , 2011 .

[15]  D. Boyd,et al.  Dynamic Debates: An Analysis of Group Polarization Over Time on Twitter , 2010 .

[16]  David J Ball,et al.  The Feeling of Risk: New Perspectives on Risk Perception , 2010 .

[17]  Fiona Fidler,et al.  Reducing Overconfidence in the Interval Judgments of Experts , 2010, Risk analysis : an official publication of the Society for Risk Analysis.

[18]  W. Aspinall.  A route to more tractable expert advice , 2010, Nature.

[19]  Hugh P. Possingham,et al.  How useful is expert opinion for predicting the distribution of a species within and beyond the region of expertise? A case study using brush-tailed rock-wallabies Petrogale penicillata , 2009 .

[20]  P. Gale,et al.  Assessing the impact of climate change on vector-borne viruses in the EU through the elicitation of expert opinion , 2009, Epidemiology and Infection.

[21]  Robert Fish,et al.  Unruly pathogens: eliciting values for environmental risk in the context of heterogeneous expert knowledge , 2009 .

[22]  R. Baayen,et al.  Mixed-effects modeling with crossed random effects for subjects and items , 2008 .

[23]  Janet M. Carey,et al.  Linguistic Uncertainty in Qualitative Risk Analysis and How to Minimize It , 2008, Annals of the New York Academy of Sciences.

[24]  M. Kynn.  The ‘heuristics and biases’ bias in expert elicitation , 2007 .

[25]  Julius Caesar,et al.  So right it ’ s wrong : Groupthink and the ubiquitous nature of polarized group decision-making , 2007 .

[26]  M. Kandlikar,et al.  Health risk assessment for nanoparticles: A case for using expert judgment , 2006 .

[27]  Jeremy E. Oakley,et al.  Uncertain Judgements: Eliciting Experts' Probabilities , 2006 .

[28]  Andy Dong,et al.  The latent semantic approach to studying design team communication , 2005 .

[29]  M. Burgman.  Risks and Decisions for Conservation and Environmental Management: Experts, stakeholders and elicitation , 2005 .

[30]  Milind Kandlikar,et al.  Representing and communicating deep uncertainty in climate-change assessments , 2005 .

[31]  N. Kerr,et al.  Group performance and decision making. , 2004, Annual review of psychology.

[32]  D. Kahneman,et al.  Heuristics and Biases: The Psychology of Intuitive Judgment , 2002 .

[33]  Helen M. Regan,et al.  A TAXONOMY AND TREATMENT OF UNCERTAINTY FOR ECOLOGY AND CONSERVATION BIOLOGY , 2002 .

[34]  E. Shevliakova,et al.  Elicitation of Expert Judgments of Climate Change Impacts on Forest Ecosystems , 2001 .

[35]  Lee H. MacDonald,et al.  Evaluating and Managing Cumulative Effects: Process and Constraints , 2000, Environmental management.

[36]  Douglas M. Bates,et al.  LINEAR AND NONLINEAR MIXED-EFFECTS MODELS , 1998 .

[37]  G Gigerenzer,et al.  Reasoning the fast and frugal way: models of bounded rationality. , 1996, Psychological review.

[38]  A. J. Underwood,et al.  Beyond BACI: Experimental designs for detecting human environmental impacts on temporal variations in natural populations , 1991 .

[39]  B. Fischhoff,et al.  Lay Foibles and Expert Fables in Judgments about Risk , 1982 .

[40]  A. Tversky,et al.  Judgment under Uncertainty: Heuristics and Biases , 1974, Science.

[41]  I. Janis.  Victims of Groupthink: A psychological study of foreign-policy decisions and fiascoes. By Irving L. Janis. (Boston: Houghton Mifflin, 1972. viii + 276 pp. Map, illustrations, chart, notes, sources, bibliography, and index. Cloth, $7.95; paper $4.50.) , 1973 .

[42]  Daniel Kahneman,et al.  Availability: A heuristic for judging frequency and probability , 1973 .