Consistency of impact assessment protocols for non-native species

Standardized tools are needed to identify and prioritize the most harmful non-native species (NNS). A plethora of assessment protocols have been developed to evaluate the current and potential impacts of non-native species, but consistency among them has received limited attention. To estimate the consistency across impact assessment protocols, 89 specialists in biological invasions used 11 protocols to screen 57 NNS (2614 assessments). We tested if the consistency in the impact scoring across assessors, quantified as the coefficient of variation (CV), was dependent on the characteristics of the protocol, the taxonomic group and the expertise of the assessor. Mean CV across assessors was 40%, with a maximum of 223%. CV was lower for protocols with a low number of score levels, which demanded high levels of expertise, and when the assessors had greater expertise on the assessed species. The similarity among protocols with respect to the final scores was higher when the protocols considered the same impact types. We conclude that all protocols led to considerable inconsistency among assessors. In order to improve consistency, we highlight the importance of selecting assessors with high expertise, providing clear guidelines and adequate training but also deriving final decisions collaboratively by consensus.

Wolfgang Nentwig | Johanna Witzell | Gritta Schrader | Helen E. Roy | Konstantinos Tsiamis | Marc Kenis | Jan Pergl | Pieter Boets | Stelios Katsanevakis | Claudia Giuliani | Mi-Jung Bae | Sonia Vanderhoeven | Hélia Marchante | Elizabete Marchante | Tim Adriaens | Lucija Šerić Jelaska | Niki Chartosia | Dan Minchin | Jonathan M. Jeschke | Franz Essl | Wolfgang Rabitsch | Toril Loennechen Moen | Bruno Foggi | Sven Bacher | Vanessa Lozano | Giuseppe Brundu | João Canning-Clode | Elena Tricarico | Pablo González-Moreno | Sylvaine Giakoumi | Cristina Preda | Predrag Simonović | Riikka Puntila | Carmen Romeralo | R. Tomov | M. Vilà | C. Giuliani | S. Giakoumi | J. Pergl | Ø. Wiig | J. Jeschke | H. Roy | B. Foggi | J. Canning‐Clode | S. Katsanevakis | S. Bacher | M. Kenis | D. Minchin | W. Nentwig | P. Boets | B. Gallardo | E. Marchante | H. Marchante | F. Lucy | V. Lozano | G. Brundu | E. García‐Berthou | W. Rabitsch | F. Essl | M. Bae | G. Schrader | A. Vettraino | P. Borges | A. Martinou | Wolf-Christian Saul | G. Copp | F. Crocetta | Hugo Verreycken | E. Tricarico | T. Adriaens | P. González-Moreno | S. Vanderhoeven | B. D’hondt | K. Tsiamis | M. Bariche | C. Preda | Ciaran Laverty | L. Lazzaro | K. Arbačiauskas | M. Auger-Rozenberg | A. Uludag | Gordon H. Copp | Leif Sundheim | Emili García-Berthou | A. Maceda-Veiga | Kęstutis Arbačiauskas | Alberto Maceda-Veiga | Montserrat Vilà | L. Vilar | Ana Montero-Castaño | N. Chartosia | Belinda Gallardo | Nikica Ogris | J. Witzell | C. Morales‐Rodrìguez | N. Ogris | M. Boieiro | Øystein Wiig | Hugo Verreycken | H. Onen | R. Sheehan | Lorenzo Lazzaro | Anna Maria Vettraino | Mário Boieiro | Carla Rego | Greta Srėbalienė | Eleni Kytinou | Angeliki F. Martinou | Frances E. Lucy | Marie-Anne Auger-Rozenberg | Bram D'hondt | Swen Follak | Ciaran Laverty | A. Tarkan | Elizabeth J. Cottier-Cook | Ali Serhan Tarkan | Wolf-Christian Saul | C. Rego | A. Soares | Venche Talgø  | Johan van Valkenburg | Rumen Tomov | Ana Andjelković | Michel Bariche | Paulo Alexandre Borges | Federico Cardigos | Elizabeth Joanne Cottier-Cook | Fabio Crocetta | Øivind Gammelmo | Fried Guillaume | Miquel Jover | Alejandro Juárez-Escario | Stefanos Kalogirou | Aleksandra Kočić | Eleni Kytinou | Sandro Meyer | Maria Cristina Morais | Carmen Morales-Rodriguez | Naida Muhthassim | Zoltán Á. Nagy | Huseyin Onen | Triya Tessa Ramburn | Fabian Reichenbach | Rory Sheehan | Marius Skolka | António Onofre Soares | Ahmet Uludağ | Lluís Vilar | Andrea Zanetta | S. Follak | A. Juárez-Escario | A. Andjelković | C. Romeralo | S. Kalogirou | M. Morais | L. Sundheim | A. Zanetta | P. Simonović | Marius Skolka | M. Jover | Øivind Gammelmo | A. Kočić | L. S. Jelaska | Greta Srėbalienė | A. Montero‐Castaño | T. T. Ramburn | F. Guillaume | Venche Talgø  | J. Van Valkenburg | T. Moen | Riikka Puntila | Zoltán Nagy | F. Reichenbach | Sandro Meyer | F. Cardigos | Naida Muhthassim | B. D'hondt | Alejandro Juárez-Escario

[1]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[2]  Sergej Olenin,et al.  Assessment of bioinvasion impacts on a regional scale: a comparative approach , 2011, Biological Invasions.

[3]  B. V. van Wilgen.  Impact of Biological Invasions on Ecosystem Services , 2017 .

[4]  Wolfgang Nentwig,et al.  Framework and guidelines for implementing the proposed IUCN Environmental Impact Classification for Alien Taxa (EICAT) , 2015 .

[5]  M. Vilà,et al.  Impact of Biological Invasions on Ecosystem Services , 2017 .

[6]  D. Richardson,et al.  Ecological Impacts of Alien Species: Quantification, Scope, Caveats, and Recommendations , 2015 .

[7]  M. G. Morgan.  Use (and abuse) of expert elicitation in support of decision making for public policy , 2014, Proceedings of the National Academy of Sciences.

[8]  Gritta Schrader,et al.  Consistency in pest risk analysis - how can it be achieved and what are the benefits?* , 2012 .

[9]  Wolfgang Nentwig,et al.  Assessing the assessments: evaluation of four impact assessment protocols for invasive alien species , 2017 .

[10]  Jaakko Heikkilä,et al.  A review of risk prioritisation schemes of pathogens, pests and weeds: principles and practices , 2008 .

[11]  H. Schielzeth.  Simple means to improve the interpretability of regression coefficients , 2010 .

[12]  Wolfgang Nentwig,et al.  A Generic Impact‐Scoring System Applied to Alien Mammals in Europe , 2010, Conservation biology : the journal of the Society for Conservation Biology.

[13]  D. Richardson,et al.  Socio‐economic impact classification of alien taxa (SEICAT) , 2018 .

[14]  Jeremy E. Oakley,et al.  Uncertain Judgements: Eliciting Experts' Probabilities , 2006 .

[15]  M. Kenis,et al.  Impact of alien terrestrial arthropods in Europe. Chapter 5 , 2010 .

[16]  Petr Pyšek,et al.  Predicting invasions by woody species in a temperate zone: a test of three risk assessment schemes in the Czech Republic (Central Europe) , 2006 .

[17]  Petr Pyšek,et al.  A proposed unified framework for biological invasions. , 2011, Trends in ecology & evolution.

[18]  R. Cousens.  Risk Assessment of Potential Biofuel Species: An Application for Trait-Based Models for Predicting Weediness , 2008, Weed Science.

[19]  V. Semenchenko,et al.  Assessing the Risks of Aquatic Species Invasions via European Inland Waterways: From Concepts to Environmental Indicators , 2009, Integrated environmental assessment and management.

[20]  M. Vilà,et al.  Comparing impacts of alien plants and animals in Europe using a standard scoring system , 2015 .

[21]  Jennifer L. Molnar,et al.  Assessing the global threat of invasive species to marine biodiversity , 2008 .

[22]  M. Vilà,et al.  Ecological impacts of invasive alien plants: a meta-analysis of their effects on species, communities and ecosystems. , 2011, Ecology letters.

[23]  M. Vilà,et al.  How well do we understand the impacts of alien species on ecosystem services? A pan-European, cross-taxa assessment , 2010 .

[24]  Lorenzo Vilizzi,et al.  Calibration of FISK, an Invasiveness Screening Tool for Nonnative Freshwater Fishes , 2009, Risk analysis : an official publication of the Society for Risk Analysis.

[25]  Lorenzo Vilizzi,et al.  Development of a generic decision-support tool for identifying potentially invasive aquatic taxa: AS-ISK , 2016 .

[26]  F. Gherardi,et al.  Calibration of FI‐ISK, an Invasiveness Screening Tool for Nonnative Freshwater Invertebrates , 2010, Risk analysis : an official publication of the Society for Risk Analysis.

[27]  Etienne Branquart,et al.  Beyond protocols: improving the reliability of expert-based risk analysis underpinning invasive species policies , 2017, Biological Invasions.

[28]  Andrew Macleod,et al.  New protocols to assess the environmental impact of pests in the EPPO decision-support scheme for pest risk analysis* , 2012 .

[29]  G. van der Velde,et al.  Inconsistencies in the risk classification of alien species and implications for risk assessment in the European Union , 2017 .

[30]  Priority invasive alien plants in the Tuscan Archipelago (Italy): comparing the EPPO prioritization scheme with the Australian WRA , 2016, Biological Invasions.

[31]  Sergej Olenin,et al.  Biological invasion impact assessment system: From idea to implementation , 2012, Ecol. Informatics.

[32]  Damaris Zurell,et al.  Collinearity: a review of methods to deal with it and a simulation study evaluating their performance , 2013 .

[33]  Els Ducheyne,et al.  Harmonia+ and Pandora+: risk screening tools for potentially invasive plants, animals and their pathogens , 2015, Biological Invasions.

[34]  D. Kahneman,et al.  Heuristics and Biases: The Psychology of Intuitive Judgment , 2002 .

[35]  Shane A. Richards,et al.  Dealing with overdispersed count data in applied ecology , 2007 .

[36]  W. Sutherland,et al.  Policy advice: Use experts wisely , 2015, Nature.

[37]  Daniel Sol,et al.  TEASIng apart alien species risk assessments: a framework for best practices. , 2012, Ecology letters.

[38]  S. Katsanevakis,et al.  Impacts of invasive alien marine species on ecosystem services and biodiversity: a pan-European review. , 2014 .

[39]  Helen E. Roy,et al.  Generic ecological impact assessments of alien species in Norway: a semi-quantitative set of criteria , 2012, Biodiversity and Conservation.

[40]  Gritta Schrader,et al.  Enhancements of pest risk analysis techniques , 2010 .

[41]  Wolfgang Nentwig,et al.  The generic impact scoring system (GISS): a standardized tool to quantify the impacts of alien species , 2016, Environmental Monitoring and Assessment.

[42]  Franz Essl,et al.  Review of risk assessment systems of IAS in Europe and introducing the German-Austrian Black List Information System (GABLIS). , 2011 .

[43]  R. Ferris,et al.  The UK risk assessment scheme for all non-native species , 2008 .

[44]  Wolfgang Nentwig,et al.  Developing a framework of minimum standards for the risk assessment of alien species , 2018 .

[45]  John D. Mumford,et al.  Invasive non-native species risk assessment in Great Britain. , 2010 .

[46]  D. Roy,et al.  Impact of alien terrestrial arthropods in Europe. , 2010 .

[47]  Sergej Olenin,et al.  Assessment of biopollution in aquatic ecosystems. , 2007, Marine pollution bulletin.

[48]  G. Copp,et al.  The Fish Invasiveness Screening Kit (FISK) for Non‐Native Freshwater Fishes—A Summary of Current Applications , 2013, Risk analysis : an official publication of the Society for Risk Analysis.

[49]  William J. Sutherland,et al.  The Delphi technique in ecology and biological conservation: applications and guidelines , 2015 .

[50]  M. Vilà,et al.  A review of impact assessment protocols of non-native plants , 2018, Biological Invasions.

[51]  P. Pheloung,et al.  A weed risk assessment model for use as a biosecurity tool evaluating plant introductions , 1999 .

[52]  Melodie A. McGeoch,et al.  Prioritizing species, pathways, and sites to achieve conservation targets for biological invasion , 2015, Biological Invasions.

[53]  Richard Baker,et al.  The EPPO prioritization process for invasive alien plants , 2010 .