An empirical workflow to integrate uncertainty and sensitivity analysis to evaluate agent-based simulation outputs

Abstract This paper presents an empirical study comparing different uncertainty analysis (UA) and sensitivity analysis (SA) methods, focussing their usefulness for the output analysis of land use/land cover change (LUCC) agent-based models (ABMs). As a result, a workflow to integrate UA and SA is presented to evaluate ABMs outputs. We developed a baseline scenario and performed a comprehensive investigation of the impacts that differences in sample sizes, sample techniques, and SA methods may have on the model output. The analysis is done in the context of a particular agent-based simulator with a LUCC model in a Brazilian Cerrado case study. The experiments indicate that there are known challenges to be overcome by the use of statistical methods. Even though the presented analysis was done over a particular simulator, we intend to contribute to the community that understands the importance of statistical validation techniques to improve the level of confidence in agent-based simulation outputs.

[1]  Maria Fonoberova,et al.  Global sensitivity/uncertainty analysis for agent-based models , 2013, Reliab. Eng. Syst. Saf..

[2]  Thomas Berger,et al.  An agent-based simulation model of human-environment interactions in agricultural systems , 2011, Environ. Model. Softw..

[3]  John Norton,et al.  An introduction to sensitivity assessment of simulation models , 2015, Environ. Model. Softw..

[4]  Paola Annoni,et al.  Sixth International Conference on Sensitivity Analysis of Model Output How to avoid a perfunctory sensitivity analysis , 2010 .

[5]  Peter H. Verburg,et al.  Simulating feedbacks in land use and land cover change models , 2006, Landscape Ecology.

[6]  Jing Wang,et al.  Parameter sensitivity analysis of crop growth models based on the extended Fourier Amplitude Sensitivity Test method , 2013, Environ. Model. Softw..

[7]  Stefano Tarantola,et al.  Sensitivity and uncertainty analysis in spatial modelling based on GIS , 2000 .

[8]  Raymond R. Hill,et al.  A Survey of Agent-Based Modeling Practices (January 1998 to July 2008) , 2009, J. Artif. Soc. Soc. Simul..

[9]  I. Sobola,et al.  Global sensitivity indices for nonlinear mathematical models and their Monte Carlo estimates , 2001 .

[10]  M. Gibbs,et al.  Efficient implementation of gaussian processes , 1997 .

[11]  Eric Koomen,et al.  Comparing the input, output, and validation maps for several models of land change , 2008 .

[12]  Marc C. Kennedy,et al.  A Bayesian sensitivity analysis applied to an Agent-based model of bird population response to landscape change , 2013, Environ. Model. Softw..

[13]  P. Soranno,et al.  Using Uncertainty and Sensitivity Analyses in Socioecological Agent-Based Models to Improve Their Analytical Performance and Policy Relevance , 2014, PloS one.

[14]  Wei-Yin Loh,et al.  Classification and regression trees , 2011, WIREs Data Mining Knowl. Discov..

[15]  Michael E. Bratman,et al.  Intention, Plans, and Practical Reason , 1991 .

[16]  Robert Gilmore Pontius,et al.  Useful techniques of validation for spatially explicit land-change models , 2004 .

[17]  Birgit Müller,et al.  A standard protocol for describing individual-based and agent-based models , 2006 .

[18]  Michael Wooldridge,et al.  Introduction to multiagent systems , 2001 .

[19]  N. Metropolis,et al.  The Monte Carlo method. , 1949 .

[20]  H. Chipman,et al.  BART: Bayesian Additive Regression Trees , 2008, 0806.3286.

[21]  A. Saltelli,et al.  Making best use of model evaluations to compute sensitivity indices , 2002 .

[22]  William F. Laurance,et al.  Increasing world consumption of beef as a driver of regional and global change: A call for policy action based on evidence from Queensland (Australia), Colombia and Brazil , 2009 .

[23]  Arika Ligmann-Zielinska,et al.  Applying time-dependent variance-based global sensitivity analysis to represent the dynamics of an agent-based model of land use change , 2010, Int. J. Geogr. Inf. Sci..

[24]  John L. Herbohn,et al.  Capital Budgeting: Financial Appraisal of Investment Projects , 2002 .

[25]  Max D. Morris,et al.  Factorial sampling plans for preliminary computational experiments , 1991 .

[26]  M. Janssen,et al.  Multi-Agent Systems for the Simulation of Land-Use and Land-Cover Change: A Review , 2003 .

[27]  Giovanni Dosi,et al.  On the robustness of the fat-tailed distribution of firm growth rates: a global sensitivity analysis , 2016 .

[28]  J. C. Helton,et al.  An Investigation of Uncertainty and Sensitivity Analysis Techniques for Computer Models , 1988 .

[29]  Carsten Peterson,et al.  Finding the Embedding Dimension and Variable Dependencies in Time Series , 1994, Neural Computation.

[30]  Joseph H. A. Guillaume,et al.  Integrated assessment and modelling: Overview and synthesis of salient dimensions , 2015, Environ. Model. Softw..

[31]  Verena Rieser,et al.  Agent-based modelling of land use dynamics and residential quality of life for future scenarios , 2013, Environ. Model. Softw..

[32]  Roman B. Statnikov,et al.  Multicriteria Analysis in Engineering , 2002 .

[33]  Stefano Tarantola,et al.  Sensitivity analysis of spatial models , 2009, Int. J. Geogr. Inf. Sci..

[34]  Ilya M. Sobol,et al.  Sensitivity Estimates for Nonlinear Mathematical Models , 1993 .

[35]  Boxin Tang.  Orthogonal Array-Based Latin Hypercubes , 1993 .

[36]  Winfried Kurth,et al.  Facilitating Parameter Estimation and Sensitivity Analysis of Agent-Based Models: A Cookbook Using NetLogo and 'R' , 2014, J. Artif. Soc. Soc. Simul..

[37]  Daniel G. Brown,et al.  Empirical characterisation of agent behaviours in socio-ecological systems , 2011, Environ. Model. Softw..

[38]  Stefano Tarantola,et al.  Sensitivity Analysis in Practice: A Guide to Assessing Scientific Models , 2004 .

[39]  Marc B. Neumann,et al.  Global sensitivity analysis for urban water quality modelling: terminology, convergence and comparison of different methods. , 2015 .

[40]  Jim W. Hall,et al.  Sensitivity analysis of environmental models: A systematic review with practical workflow , 2014, Environ. Model. Softw..

[41]  Anthony J. Jakeman,et al.  Selecting among five common modelling approaches for integrated environmental assessment and management , 2013, Environ. Model. Softw..

[42]  Wei Chu,et al.  A comprehensive evaluation of various sensitivity analysis methods: A case study with a hydrological model , 2014, Environ. Model. Softw..

[43]  Winfried Lamersdorf,et al.  Jadex: A BDI-Agent System Combining Middleware and Reasoning , 2005 .

[44]  Matthias Meyer,et al.  Opening the ‘black box’ of simulations: increased transparency and effective communication through the systematic design of experiments , 2011, Computational and Mathematical Organization Theory.

[45]  C. Fortuin,et al.  Study of the sensitivity of coupled reaction systems to uncertainties in rate coefficients. I Theory , 1973 .

[46]  Célia Ghedini Ralha,et al.  MASE-BDI: agent-based simulator for environmental land change with efficient and parallel auto-tuning , 2016, Applied Intelligence.

[47]  Rob A. Rutenbar,et al.  Why Quasi-Monte Carlo is Better Than Monte Carlo or Latin Hypercube Sampling for Statistical Circuit Analysis , 2010, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems.

[48]  C. Spearman.  The proof and measurement of association between two things. By C. Spearman, 1904. , 1987, The American journal of psychology.

[49]  W. Parton,et al.  Land use change: complexity and comparisons , 2008, Journal of land use science.

[50]  V. Grimm.  Ten years of individual-based modelling in ecology: what have we learned and what could we learn in the future? , 1999 .

[51]  M. Winograd,et al.  Dynamics of the agricultural frontier in the Amazon and savannas of Brazil: analyzing the impact of policy and technology , 1998 .

[52]  Saltelli Andrea,et al.  Global Sensitivity Analysis: The Primer , 2008 .

[53]  Florian Pappenberger,et al.  Multi-method global sensitivity analysis of flood inundation models. , 2008 .

[54]  Célia Ghedini Ralha,et al.  A multi-agent model system for land-use change simulation , 2013, Environ. Model. Softw..

[55]  Behrooz Hassani-Mahmooei,et al.  "Anarchy" Reigns: A Quantitative Analysis of Agent-Based Modelling Publication Practices in JASSS, 2001-2012 , 2015, J. Artif. Soc. Soc. Simul..

[56]  Forrest Stonedahl,et al.  The Complexities of Agent-Based Modeling Output Analysis , 2015, J. Artif. Soc. Soc. Simul..

[57]  Jack P. C. Kleijnen,et al.  State-of-the-Art Review: A User's Guide to the Brave New World of Designing Simulation Experiments , 2005, INFORMS J. Comput..

[58]  Wei Chen,et al.  Assessing the Reliability of Complex Models: Mathematical and Statistical Foundations of Verification, Validation, and Uncertainty Quantification , 2012 .

[59]  J. Friedman.  Multivariate adaptive regression splines , 1990 .

[60]  Arend Ligtenberg,et al.  Which Sensitivity Analysis Method Should I Use for My Agent-Based Model? , 2016, J. Artif. Soc. Soc. Simul..

[61]  Peter Zinterhof,et al.  Monte Carlo and Quasi-Monte Carlo Methods 1996 , 1998 .

[62]  Jing Yang,et al.  Convergence and uncertainty analyses in Monte-Carlo based sensitivity analysis , 2011, Environ. Model. Softw..

[63]  Wei Gong,et al.  An Intercomparison of Sampling Methods for Uncertainty Quantification of Environmental Dynamic Models , 2015 .

[64]  M. D. McKay,et al.  A comparison of three methods for selecting values of input variables in the analysis of output from a computer code , 2000 .

[65]  Averill M. Law,et al.  Simulation Modeling and Analysis , 1982 .

[66]  P. Reed,et al.  Hydrology and Earth System Sciences Discussions Comparing Sensitivity Analysis Methods to Advance Lumped Watershed Model Identification and Evaluation , 2022 .