Meta-models as a straightforward approach to the sensitivity analysis of complex models

Complex simulation models are important tools in applied ecological and conservation research. However sensitivity analysis of this important class of models can be difficult to conduct. High level interactions and non-linear responses are common in complex simulations, and this necessitates a global sensitivity analysis, where each parameter is tested at a range of values, and in combination with changes in many other parameters. We reviewed the literature, searching for population viability analyses that used simulation models. We found only 9 out of the 122 simulation population viability analysis used global sensitivity analysis. This result is typical of other simulation models in applied ecology, where global sensitivity analysis is rare. We then demonstrate how to conduct a meta-modeling sensitivity analysis, where a simpler statistically fit function (the meta-model, also known as the surrogate model or emulator) is used to approximate the behavior of the complicated simulation. This simpler meta-model is interrogated to inform on the behavior of simulation model. We fit two example meta-models, a generalized linear model and a boosted regression tree, to exemplify the approach. Our hope is that by going through these techniques thoroughly they will become more widely adopted.

[1]  Michael E. Tewes,et al.  Evaluating recovery strategies for an ocelot (Leopardus pardalis) population in the united states , 2005 .

[2]  J. Friedman.  Greedy function approximation: A gradient boosting machine. , 2001 .

[3]  M. Hebblewhite,et al.  The Role of Translocation in Recovery of Woodland Caribou Populations , 2010, Conservation biology : the journal of the Society for Conservation Biology.

[4]  J Elith,et al.  A working guide to boosted regression trees. , 2008, The Journal of animal ecology.

[5]  X. Y. Sun,et al.  Three complementary methods for sensitivity analysis of a water quality model , 2012, Environ. Model. Softw..

[6]  J M R Curtis,et al.  Sensitivity of population viability to spatial and nonspatial parameters using GRIP. , 2008, Ecological applications : a publication of the Ecological Society of America.

[7]  Jon C. Helton,et al.  Implementation and evaluation of nonparametric regression procedures for sensitivity analysis of computationally demanding models , 2009, Reliab. Eng. Syst. Saf..

[8]  Jeremy Rohmer,et al.  Global sensitivity analysis of large-scale numerical landslide models based on Gaussian-Process meta-modeling , 2011, Comput. Geosci..

[9]  R. Gramacy,et al.  Categorical Inputs, Sensitivity Analysis, Optimization and Importance Tempering with tgp Version 2, an R Package for Treed Gaussian Process Models , 2010 .

[10]  A. Crivelli,et al.  Potential factors controlling the population viability of newly introduced endangered marble trout populations , 2008 .

[11]  David B. Lindenmayer,et al.  A simulation study of the impacts of population subdivision on the mountain brushtail possum Trichosurus caninus Ogilby (Phalangeridae: Marsupialia), in south-eastern Australia. II. Loss of genetic variation within and between subpopulations , 1995 .

[12]  D. T. Briese,et al.  Demography and management of the invasive plant species Hypericum perforatum. II. Construction and use of an individual‐based model to predict population dynamics and the effects of management strategies , 2003 .

[13]  S. Ellner,et al.  USING PVA FOR MANAGEMENT DESPITE UNCERTAINTY: EFFECTS OF HABITAT, HATCHERIES, AND HARVEST ON SALMON , 2003 .

[14]  R. Brandl,et al.  Viability analysis of endangered crayfish populations , 2007 .

[15]  Kimberly S. Sheldon,et al.  On a collision course: competition and dispersal differences create no-analogue communities and cause extinctions during climate change , 2012, Proceedings of the Royal Society B: Biological Sciences.

[16]  M. WallisDeVries,et al.  Using surrogate data in population viability analysis: the case of the critically endangered cranberry fritillary butterfly , 2005 .

[17]  M. Bode,et al.  Using population viability analysis to guide research and conservation actions for Australia's threatened malleefowl Leipoa ocellata , 2011, Oryx.

[18]  Paul C. Cross,et al.  Using Logistic Regression to Analyze the Sensitivity of PVA Models: a Comparison of Methods Based on African Wild Dog Models , 2001 .

[19]  N. Schtickzelle,et al.  Metapopulation dynamics of the bog fritillary butterfly: modelling the effect of habitat fragmentation , 2002 .

[20]  D. Macdonald,et al.  Trapping for mink control and water vole survival : Identifying key criteria using a spatially explicit individual based model , 2007 .

[21]  Andrea Castelletti,et al.  Emulation techniques for the reduction and sensitivity analysis of complex environmental models , 2012, Environ. Model. Softw..

[22]  Scott Ferson,et al.  Sensitivity analysis for models of population viability , 1995 .

[23]  Jérémy Rohmer,et al.  Global sensitivity analysis of large-scale numerical , 2011 .

[24]  W. Ricker.  Stock and Recruitment , 1954 .

[25]  Michelle M McClure,et al.  The Interplay between Climate Variability and Density Dependence in the Population Viability of Chinook Salmon , 2006, Conservation biology : the journal of the Society for Conservation Biology.

[26]  A. Saltelli,et al.  A quantitative model-independent method for global sensitivity analysis of model output , 1999 .

[27]  Hugh P. Possingham,et al.  Reliability of Relative Predictions in Population Viability Analysis , 2003 .

[28]  Shaun R. Coutts,et al.  What are the key drivers of spread in invasive plants: dispersal, demography or landscape: and how can we use this knowledge to aid management? , 2011, Biological Invasions.

[29]  K. Kunkel,et al.  Estimating Sustainable Harvest in Wolverine Populations Using Logistic Regression , 2008 .

[30]  Emanuele Borgonovo,et al.  Model emulation and moment-independent sensitivity analysis: An application to environmental modelling , 2012, Environ. Model. Softw..

[31]  Todd R. Lookingbill,et al.  Combining a dispersal model with network theory to assess habitat connectivity. , 2010, Ecological applications : a publication of the Ecological Society of America.

[32]  C. Hunter,et al.  Parameter uncertainty and elasticity analyses of a population model: setting research priorities for shearwaters , 2000 .

[33]  Evan H. Girvetz,et al.  Removing erosion control projects increases bank swallow (Riparia riparia) population viability modeled along the Sacramento River, California, USA. , 2010 .

[34]  Jack P. C. Kleijnen,et al.  Sensitivity analysis of censored output through polynomial, logistic, and tobit regression meta-models: theory and case study , 2001, Proceeding of the 2001 Winter Simulation Conference (Cat. No.01CH37304).

[35]  Andrea Saltelli,et al.  Guest editorial: The role of sensitivity analysis in the corroboration of models and itslink to model structural and parametric uncertainty , 1997 .

[36]  Leah R. Johnson,et al.  Implications of dispersal and life history strategies for the persistence of Linyphiid spider populations , 2009, 0908.2778.