A Practical Guide for Conducting Calibration and Decision-Making Optimisation with Complex Ecological Models

Calibrating ecological models or making decisions with them is an optimisation problem with challenging methodological issues. Depending on the optimisation formulation, there may be a large variety of optimisation configurations (e.g. multiple objectives, constraints, stochastic criteria) and finding a single acceptable solution may be difficult. The challenges are exacerbated by the high computational cost and the non linear or elusive mathematical properties that increased with the complexity of numerical models. From the feedbacks of practitioners, the need for a guideline for conducting optimisation of complex models has emerged. In this context, we propose a practical guide for the complex model optimisation process, covering both calibration and decision-making. The guide sets out the workflow with recommendations for each step based on existing tools and methods usually scattered throughout the literature. This guide is accompanied with an ODDO template (Overview, Design, Details of Optimisation) to standardise the published description of model-based optimisation and suggests research directions.

[1]  Demetris Koutsoyiannis,et al.  One decade of multi-objective calibration approaches in hydrological modelling: a review , 2010 .

[2]  Bruce Edmonds,et al.  From KISS to KIDS - An 'Anti-simplistic' Modelling Approach , 2004, MABS.

[3]  Trevor Hastie,et al.  The Elements of Statistical Learning , 2001 .

[4]  Peter J. Fleming,et al.  Pareto Front Estimation for Decision Making , 2014, Evolutionary Computation.

[5]  Roger D Peng,et al.  Reproducible research and Biostatistics. , 2009, Biostatistics.

[6]  Perry de Valpine,et al.  Review of methods for fitting time-series models with process and observation error and likelihood calculations for nonlinear, non-Gaussian state-space models , 2002 .

[7]  Sara E. Miller,et al.  Data conflicts in fishery models: incorporating hydroacoustic data into the Prince William Sound Pacific herring assessment model , 2008 .

[8]  Guillaume Deffuant,et al.  Adaptive approximate Bayesian computation for complex models , 2011, Computational Statistics.

[9]  Jenný Brynjarsdóttir,et al.  Learning about physical parameters: the importance of model discrepancy , 2014 .

[10]  Kalyanmoy Deb,et al.  Reference point based multi-objective optimization using evolutionary algorithms , 2006, GECCO.

[11]  John R. Skalski,et al.  Variance estimation in integrated assessment models and its importance for hypothesis testing , 2007 .

[12]  Bertrand Iooss,et al.  Global sensitivity analysis of stochastic computer models with joint metamodels , 2008, Statistics and Computing.

[13]  Jack P. C. Kleijnen,et al.  A methodology for the fitting and validation of metamodels in simulation , 2000 .

[14]  Steven F. Railsback,et al.  Chapter 11. Using Analytical Models in Individual-based Ecology , 2005 .

[15]  Shumeet Baluja,et al.  A Method for Integrating Genetic Search Based Function Optimization and Competitive Learning , 1994 .

[16]  Patrick Siarry,et al.  A survey on optimization metaheuristics , 2013, Inf. Sci..

[17]  M. Powell A Direct Search Optimization Method That Models the Objective and Constraint Functions by Linear Interpolation , 1994 .

[18]  Nikolaos V. Sahinidis,et al.  Derivative-free optimization: a review of algorithms and comparison of software implementations , 2013, J. Glob. Optim..

[19]  Nikolaos V. Sahinidis,et al.  Simulation optimization: a review of algorithms and applications , 2014, 4OR.

[20]  Byron J. T. Morgan,et al.  Methods for investigating parameter redundancy , 2004 .

[21]  Michèle Sebag,et al.  Intensive surrogate model exploitation in self-adaptive surrogate-assisted cma-es (saacm-es) , 2013, GECCO '13.

[22]  Holger H. Hoos,et al.  Automated Algorithm Configuration and Parameter Tuning , 2012, Autonomous Search.

[23]  Stéphanie Mahévas,et al.  Analyse de sensibilité et exploration de modèles , 2013 .

[24]  Stéphanie Mahévas,et al.  Supporting Fisheries Management by Means of Complex Models: Can We Point out Isles of Robustness in a Sea of Uncertainty? , 2013, PloS one.

[25]  David A. Fournier,et al.  MULTIFAN a Likelihood-Based Method for Estimating Growth Parameters and Age Composition from Multiple Length Frequency Data Sets Illustrated using Data for Southern Bluefin Tuna (Thunnus maccoyii) , 1990 .

[26]  José Mario Martínez,et al.  Algorithm 813: SPG—Software for Convex-Constrained Optimization , 2001, TOMS.

[27]  Paul Fearnhead,et al.  Constructing Summary Statistics for Approximate Bayesian Computation: Semi-automatic ABC , 2010, 1004.1112.

[28]  Carlos Artemio Coello-Coello,et al.  Theoretical and numerical constraint-handling techniques used with evolutionary algorithms: a survey of the state of the art , 2002 .

[29]  R. Levins The strategy of model building in population biology , 1966 .

[30]  Bryan A. Tolson,et al.  A benchmarking framework for simulation-based optimization of environmental models , 2012, Environ. Model. Softw..

[31]  C. D. Perttunen,et al.  Lipschitzian optimization without the Lipschitz constant , 1993 .

[32]  Arnold Neumaier,et al.  Global Optimization by Multilevel Coordinate Search , 1999, J. Glob. Optim..

[33]  R. K. Ursem Multi-objective Optimization using Evolutionary Algorithms , 2009 .

[34]  Peter J. Fleming,et al.  Many-Objective Optimization: An Engineering Design Perspective , 2005, EMO.

[35]  Jorge Nocedal,et al.  On the limited memory BFGS method for large scale optimization , 1989, Math. Program..

[36]  John A. Nelder,et al.  A Simplex Method for Function Minimization , 1965, Comput. J..

[37]  François M. Hemez,et al.  Info-gap robustness for the correlation of tests and simulations of a non-linear transient , 2004 .

[38]  Stéphanie Mahévas,et al.  Reconciling complex system models and fisheries advice: Practical examples and leads , 2016 .

[39]  Petros Koumoutsakos,et al.  Reducing the Time Complexity of the Derandomized Evolution Strategy with Covariance Matrix Adaptation (CMA-ES) , 2003, Evolutionary Computation.

[40]  Hadi Dowlatabadi,et al.  Sensitivity and Uncertainty Analysis of Complex Models of Disease Transmission: an HIV Model, as an Example , 1994 .

[41]  Uta Berger,et al.  Pattern-Oriented Modeling of Agent-Based Complex Systems: Lessons from Ecology , 2005, Science.

[42]  Christophe Sibertin-Blanc,et al.  Calibration of simulation platforms including highly interweaved processes: the MAELIA multi-agent platform , 2014 .

[43]  Nicola Beume,et al.  SMS-EMOA: Multiobjective selection based on dominated hypervolume , 2007, Eur. J. Oper. Res..

[44]  Michaël Dingkuhn,et al.  Compensatory phenotypic plasticity in irrigated rice: Sequential formation of yield components and simulation with SAMARA model , 2016 .

[45]  Avi Ostfeld,et al.  Evolutionary algorithms and other metaheuristics in water resources: Current status, research challenges and future directions , 2014, Environ. Model. Softw..

[46]  R. Horst,et al.  Global Optimization: Deterministic Approaches , 1992 .

[47]  Maarten Keijzer,et al.  Evolving Objects: A General Purpose Evolutionary Computation Library , 2001, Artificial Evolution.

[48]  Stéphanie Mahévas,et al.  ISIS-Fish, a generic and spatially explicit simulation tool for evaluating the impact of management measures on fisheries dynamics , 2004 .

[49]  Anne Auger,et al.  Comparing results of 31 algorithms from the black-box optimization benchmarking BBOB-2009 , 2010, GECCO '10.

[50]  Pedro Larrañaga,et al.  Estimation of Distribution Algorithms , 2002, Genetic Algorithms and Evolutionary Computation.

[51]  Victor Picheny,et al.  Optimization of black-box models with uncertain climatic inputs—Application to sunflower ideotype design , 2017, PloS one.

[52]  John Sibert,et al.  AD Model Builder: using automatic differentiation for statistical inference of highly parameterized complex nonlinear models , 2012, Optim. Methods Softw..

[53]  M. Powell The NEWUOA software for unconstrained optimization without derivatives , 2006 .

[54]  L. Lebart,et al.  Statistique exploratoire multidimensionnelle , 1995 .

[55]  J. Gareth Polhill,et al.  The ODD protocol: A review and first update , 2010, Ecological Modelling.

[56]  Tobias Glasmachers,et al.  Anytime Bi-Objective Optimization with a Hybrid Multi-Objective CMA-ES (HMO-CMA-ES) , 2016, GECCO.

[57]  Fernando G. Lobo,et al.  A parameter-less genetic algorithm , 1999, GECCO.

[58]  Oswin Krause,et al.  Unbounded Population MO-CMA-ES for the Bi-Objective BBOB Test Suite , 2016, GECCO.

[59]  Joseph R. Kasprzyk,et al.  Many objective robust decision making for complex environmental systems undergoing change , 2012, Environ. Model. Softw..

[60]  Pierre Petitgas,et al.  A simulation-based approach to assess sensitivity and robustness of fisheries management indicators for the pelagic fishery in the Bay of Biscay , 2013 .

[61]  John C. Nash,et al.  Strategies for fitting nonlinear ecological models in R, AD Model Builder, and BUGS , 2013 .

[62]  A. J. Booker,et al.  A rigorous framework for optimization of expensive functions by surrogates , 1998 .

[63]  Marco Antonio Luersen,et al.  Globalized Nelder-Mead method for engineering optimization , 2002 .

[64]  Arnold Neumaier,et al.  SNOBFIT -- Stable Noisy Optimization by Branch and Fit , 2008, TOMS.

[65]  N. Hansen A CMA-ES for Mixed-Integer Nonlinear Optimization , 2011 .

[66]  Birgit Müller,et al.  A standard protocol for describing individual-based and agent-based models , 2006 .

[67]  O. François,et al.  Approximate Bayesian Computation (ABC) in practice. , 2010, Trends in ecology & evolution.

[68]  Hisao Ishibuchi,et al.  Evolutionary many-objective optimization , 2008, 2008 3rd International Workshop on Genetic and Evolving Systems.

[69]  Thomas Stützle,et al.  Evaluating Las Vegas Algorithms: Pitfalls and Remedies , 1998, UAI.

[70]  Perry de Valpine,et al.  Fitting complex population models by combining particle filters with Markov chain Monte Carlo. , 2012, Ecology.

[71]  Guillaume Deffuant,et al.  The GR3D model, a tool to explore the Global Repositioning Dynamics of Diadromous fish Distribution , 2014 .

[72]  J.-M. Marin,et al.  Relevant statistics for Bayesian model choice , 2011, 1110.4700.

[73]  James D. Nichols,et al.  ESTIMATING STATE-TRANSITION PROBABILITIES FOR UNOBSERVABLE STATES USING CAPTURE–RECAPTURE/RESIGHTING DATA , 2002 .

[74]  P. McCullagh,et al.  Generalized Linear Models , 1984 .

[75]  S. Sisson,et al.  A comparative review of dimension reduction methods in approximate Bayesian computation , 2012, 1202.3819.

[76]  Carlos M. Fonseca,et al.  Inferential Performance Assessment of Stochastic Optimisers and the Attainment Function , 2001, EMO.

[77]  Donald R. Jones,et al.  A Taxonomy of Global Optimization Methods Based on Response Surfaces , 2001, J. Glob. Optim..

[78]  Maureen A. O’Malley,et al.  Do simple models lead to generality in ecology? , 2013, Trends in ecology & evolution.

[79]  Charles Audet,et al.  Mesh Adaptive Direct Search Algorithms for Constrained Optimization , 2006, SIAM J. Optim..

[80]  Nando de Freitas,et al.  A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning , 2010, ArXiv.

[81]  Virginie Le Rolle,et al.  Sensitivity Analysis and Parameter Estimation of a Coronary Circulation Model for Triple-Vessel Disease , 2014, IEEE Transactions on Biomedical Engineering.

[82]  FrancisR.I.C. Chris,et al.  Data weighting in statistical fisheries stock assessment models , 2011 .

[83]  Steven L. Lytinen,et al.  Agent-based Simulation Platforms: Review and Development Recommendations , 2006, Simul..

[84]  Wolfgang Ponweiser,et al.  Multiobjective Optimization on a Limited Budget of Evaluations Using Model-Assisted -Metric Selection , 2008, PPSN.

[85]  Kevin R. Piner,et al.  Dealing with data conflicts in statistical inference of population assessment models that integrate information from multiple diverse data sets , 2017 .

[86]  Kalyanmoy Deb,et al.  A fast and elitist multiobjective genetic algorithm: NSGA-II , 2002, IEEE Trans. Evol. Comput..

[87]  Anne Auger,et al.  COCO: a platform for comparing continuous optimizers in a black-box setting , 2016, Optim. Methods Softw..