Merging validation and evaluation of ecological models to ‘evaludation’: A review of terminology and a practical approach

Confusion about model validation is one of the main challenges in using ecological models for decision support, such as the regulation of pesticides. Decision makers need to know whether a model is a sufficiently good representation of its real counterpart and what criteria can be used to answer this question. Unclear terminology is one of the main obstacles to a good understanding of what model validation is, how it works, and what it can deliver. Therefore, we performed a literature review and derived a standard set of terms. ‘Validation’ was identified as a catch-all term, which is thus useless for any practical purpose. We introduce the term ‘evaludation’, a fusion of ‘evaluation’ and ‘validation’, to describe the entire process of assessing a model's quality and reliability. Considering the iterative nature of model development, the modelling cycle, we identified six essential elements of evaludation: (i) ‘data evaluation’ for scrutinising the quality of numerical and qualitative data used for model development and testing; (ii) ‘conceptual model evaluation’ for examining the simplifying assumptions underlying a model's design; (iii) ‘implementation verification’ for testing the model's implementation in equations and as a computer programme; (iv) ‘model output verification’ for comparing model output to data and patterns that guided model design and were possibly used for calibration; (v) ‘model analysis’ for exploring the model's sensitivity to changes in parameters and process formulations to make sure that the mechanistic basis of main behaviours of the model has been well understood; and (vi) ‘model output corroboration’ for comparing model output to new data and patterns that were not used for model development and parameterisation. Currently, most decision makers require ‘validating’ a model by testing its predictions with new experiments or data. Despite being desirable, this is neither sufficient nor necessary for a model to be useful for decision support. We believe that the proposed set of terms and its relation to the modelling cycle can help to make quality assessments and reality checks of ecological models more comprehensive and transparent.

[1]  Nassim Nicholas Taleb,et al.  The Black Swan: The Impact of the Highly Improbable , 2007 .

[2]  D. Botkin.  Forest Dynamics: An Ecological Model , 1993 .

[3]  Saul I. Gass,et al.  Feature Article - Decision-Aiding Models: Validation, Assessment, and Related Issues for Policy Analysis , 1983, Oper. Res..

[4]  Linda Pilkey-Jarvis,et al.  :Useless Arithmetic: Why Environmental Scientists Cant Predict the Future , 2007 .

[5]  James S. Hodges,et al.  Is It You or Your Model Talking?: A Framework for Model Validation , 1992 .

[6]  Robert G. Sargent,et al.  Verification and validation of simulation models , 2009, IEEE Engineering Management Review.

[7]  Lael Parrott,et al.  Levels of emergence in individual based models: Coping with scarcity of data and pattern redundancy , 2011 .

[8]  Hal Caswell,et al.  12 – The Validation Problem , 1976 .

[9]  Volker Grimm,et al.  Integrating population modeling into ecological risk assessment , 2010, Integrated environmental assessment and management.

[10]  K. Jensen,et al.  The importance of alternative conceptual models for simulation of concentrations in a multi-aquifer system , 2007 .

[11]  D. Weed.  Weight of Evidence: A Review of Concept and Methods , 2005, Risk analysis : an official publication of the Society for Risk Analysis.

[12]  V. Grimm,et al.  Post-Hoc Pattern-Oriented Testing and Tuning of an Existing Large Model: Lessons from the Field Vole , 2012, PloS one.

[13]  N Oreskes,et al.  Verification, Validation, and Confirmation of Numerical Models in the Earth Sciences , 1994, Science.

[14]  Jean-Louis Malouin,et al.  Model validation in operations research , 1983 .

[15]  V. Grimm,et al.  Ecological Models in Support of Regulatory Risk Assessments of Pesticides: Developing a Strategy for the Future , 2009, Integrated environmental assessment and management.

[16]  R. Levins.  The strategy of model building in population biology , 1966 .

[17]  Anthony J. Jakeman,et al.  Good Modelling Practice , 2008 .

[18]  J. Bart.  Acceptance Criteria for Using Individual‐Based Models to Make Management Decisions , 1995 .

[19]  T. Kuhn,et al.  The Structure of Scientific Revolutions. , 1964 .

[20]  Kurt Jax,et al.  Concepts, not terms , 2008 .

[21]  Christian Wissel,et al.  Reconstructing spatiotemporal dynamics of Central European natural beech forests: the rule-based forest model BEFORE , 2004 .

[22]  Craig A. Aumann,et al.  A methodology for developing simulation models of complex systems , 2007 .

[23]  Keith Beven,et al.  A manifesto for the equifinality thesis , 2006 .

[24]  Anthony J. Jakeman,et al.  Ten iterative steps in development and evaluation of environmental models , 2006, Environ. Model. Softw..

[25]  J D Sterman.  The meaning of models. , 1994, Science.

[26]  Flaminio Squazzoni,et al.  Epistemological Aspects of Computer Simulation in the Social Sciences, Second International Workshop, EPOS 2006, Brescia, Italy, October 5-6, 2006, Revised Selected and Invited Papers , 2009, EPOS.

[27]  Keith Beven,et al.  Towards an alternative blueprint for a physically based digitally simulated hydrologic response modelling system , 2002 .

[28]  A. Hunka,et al.  Stakeholders’ Perspective on Ecological Modeling in Environmental Risk Assessment of Pesticides: Challenges and Opportunities , 2013, Risk analysis : an official publication of the Society for Risk Analysis.

[29]  Joseph H. A. Guillaume,et al.  Characterising performance of environmental models , 2013, Environ. Model. Softw..

[30]  M. Kendall,et al.  The Logic of Scientific Discovery. , 1959 .

[31]  L. Shawn Matott,et al.  Evaluating uncertainty in integrated environmental models: A review of concepts and tools , 2009 .

[32]  M. B. Beck,et al.  On the problem of model validation for predictive exposure assessments , 1997 .

[33]  Franziska Klügl,et al.  A validation methodology for agent-based simulations , 2008, SAC '08.

[34]  Yaman Barlas,et al.  Philosophical roots of model validation: Two paradigms , 1990 .

[35]  P. J. Van den Brink,et al.  Potential application of population models in the European ecological risk assessment of chemicals II: Review of models and their potential to address environmental protection aims , 2010, Integrated environmental assessment and management.

[36]  Yaman Barlas,et al.  Formal aspects of model validity and validation in system dynamics , 1996 .

[37]  Stefano Tarantola,et al.  Sensitivity Analysis as an Ingredient of Modeling , 2000 .

[38]  David J. Mladenoff,et al.  Increasing the reliability of ecological models using modern software engineering techniques , 2010 .

[39]  Denis Borenstein,et al.  Towards a practical method to validate decision support systems , 1998, Decis. Support Syst..

[40]  Scott Ferson,et al.  Automated quality assurance checks on model structure in ecological risk assessments , 1996 .

[41]  Steven F Railsback,et al.  Pattern-oriented modelling: a ‘multi-scope’ for predictive systems ecology , 2012, Philosophical Transactions of the Royal Society B: Biological Sciences.

[42]  W. Oberkampf,et al.  Model validation and predictive capability for the thermal challenge problem , 2008 .

[43]  H. Shugart.  A Theory of Forest Dynamics , 1984 .

[44]  Robert M. Argent,et al.  An overview of model integration for environmental applications--components, frameworks and semantics , 2004, Environ. Model. Softw..

[45]  Ayalew Kassahun,et al.  Quality assurance in model based water management - review of existing practice and outline of new approaches , 2005, Environ. Model. Softw..

[46]  Ronald F. Perry,et al.  Simulation: A Problem-Solving Approach , 1989 .

[47]  K. E. Hodges,et al.  Defining the problem: terminology and progress in ecology , 2008 .

[48]  Peter Chapman,et al.  Ecological models and pesticide risk assessment: Current modeling practice , 2010, Environmental toxicology and chemistry.

[49]  C. S. Holling.  Adaptive Environmental Assessment and Management , 2005 .

[50]  Jens Christian Refsgaard,et al.  Modelling guidelinesterminology and guiding principles , 2004 .

[51]  James D. Arthur,et al.  Evaluating the Effectiveness of Independendent Verification and Validation , 1999, Computer.

[52]  Uta Berger,et al.  Pattern-Oriented Modeling of Agent-Based Complex Systems: Lessons from Ecology , 2005, Science.

[53]  S. Funtowicz,et al.  Combining Quantitative and Qualitative Measures of Uncertainty in Model‐Based Environmental Assessment: The NUSAP System , 2005, Risk analysis : an official publication of the Society for Risk Analysis.

[54]  Steven F. Railsback,et al.  Individual-based modeling and ecology , 2005 .

[55]  James S. Hodges,et al.  Six (Or So) Things You Can Do with a Bad Model , 1991, Oper. Res..

[56]  C. S. Holling.  The functional response of invertebrate predators to prey density , 1966 .

[57]  Jeroen P. van der Sluijs,et al.  A framework for dealing with uncertainty due to model structure error , 2004 .

[58]  Volker Grimm,et al.  Ecological models supporting environmental decision making: a strategy for the future. , 2010, Trends in ecology & evolution.

[59]  T. Fagerström.  On theory, data and mathematics in ecology , 1987 .

[60]  Robert Luttik,et al.  Population level risk assessment: practical considerations for evaluation of population models from a risk assessor's perspective , 2012, Environmental Sciences Europe.

[61]  Christian Wissel,et al.  Was charakterisiert Buchenurwälder? Untersuchungen der Altersstruktur des Kronendachs und der räumlichen Verteilung der Baumriesen in einem Modellwald mit Hilfe des Simulationsmodells BEFORE , 2001, Forstwissenschaftliches Centralblatt vereinigt mit Tharandter forstliches Jahrbuch.

[62]  Zsuzsanna Bacsi,et al.  Validation: an objective or a tool? Results on a winter wheat simulation model application , 1995 .

[63]  Scott Ferson,et al.  Ecological modeling in risk assessment : chemical effects on populations, ecosystems, and landscapes , 2001 .

[64]  V. Grimm,et al.  Uncertainty in predictions of range dynamics: black grouse climbing the Swiss Alps , 2012 .

[65]  Bridget Rosewell,et al.  Validation and Verification of Agent-Based Models in the Social Sciences , 2009, EPOS.

[66]  Edward J. Rykiel,et al.  Testing ecological models: the meaning of validation , 1996 .

[67]  Roman Ashauer,et al.  CREAM: a European project on mechanistic effect models for ecological risk assessment of chemicals , 2009, Environmental science and pollution research international.

[68]  George Loizou,et al.  Development of good modelling practice for physiologically based pharmacokinetic models for use in risk assessment: the first steps. , 2008, Regulatory toxicology and pharmacology : RTP.

[69]  Richard P. Hooper,et al.  Testing and validating environmental models , 1996 .

[70]  David J. Mladenoff,et al.  An ecological classification of forest landscape simulation models: tools and strategies for understanding broad-scale forested ecosystems , 2007, Landscape Ecology.

[71]  Caleb A. Oriade,et al.  Developments in biophysical and bioeconomic simulation of agricultural systems: a review , 1997 .

[72]  James S. Clark,et al.  Why environmental scientists are becoming Bayesians , 2004 .

[73]  Gunnar Abrahamson,et al.  Terminology for model credibility , 1980 .

[74]  A. Saltelli,et al.  Sensitivity Anaysis as an Ingredient of Modeling , 2000 .

[75]  J.J.T.I. Boesten,et al.  Modeller subjectivity in estimating pesticide parameters for leaching models using the same laboratory data set , 2000 .

[76]  John D. Aber,et al.  Why Don't We Believe the Models? , 1997, The Bulletin of the Ecological Society of America.