Evaluating uncertainty in integrated environmental models: A review of concepts and tools

This paper reviews concepts for evaluating integrated environmental models and discusses a list of relevant software‐based tools. A simplified taxonomy for sources of uncertainty and a glossary of key terms with “standard” definitions are provided in the context of integrated approaches to environmental assessment. These constructs provide a reference point for cataloging 65 different model evaluation tools. Each tool is described briefly (in the auxiliary material) and is categorized for applicability across seven thematic model evaluation methods. Ratings for citation count and software availability are also provided, and a companion Web site containing download links for tool software is introduced. The paper concludes by reviewing strategies for tool interoperability and offers guidance for both practitioners and tool developers.

[1]  Brian E. Skahill,et al.  More efficient PEST compatible model independent model calibration , 2009, Environ. Model. Softw..

[2]  Johnny Wei-Bing Lin.  qtcm 0.1.2: a Python implementation of the Neelin-Zeng Quasi-Equilibrium Tropical Circulation Model , 2008 .

[3]  David P Ahlfeld,et al.  Impact of Simulation Model Solver Performance on Ground Water Management Problems , 2008, Ground water.

[4]  Courtney A. Schultz.  Responding to scientific uncertainty in U.S. forest policy , 2008 .

[5]  Gemma Manache,et al.  Identification of reliable regression- and correlation-based sensitivity measures for importance ranking of water-quality model parameters , 2008, Environ. Model. Softw..

[6]  L. Matott,et al.  Calibration of subsurface batch and reactive-transport models involving complex biogeochemical processes , 2008 .

[7]  Stefano Tarantola,et al.  Variance‐Based Methods , 2008 .

[8]  Helena Mitasova,et al.  Chapter Ten Free and Open Source Geospatial Tools for Environmental Modelling and Management , 2008 .

[9]  Robert N. Stewart,et al.  Chapter Twelve Building a Community Modelling and Information Sharing Culture , 2008 .

[10]  S. P. Neuman,et al.  On model selection criteria in multimodel analysis , 2007 .

[11]  Peter A. Vanrolleghem,et al.  Uncertainty in the environmental modelling process - A framework and guidance , 2007, Environ. Model. Softw..

[12]  Andrea Saltelli,et al.  An effective screening design for sensitivity analysis of large models , 2007, Environ. Model. Softw..

[13]  J. B. Gregersen,et al.  OpenMI: Open modelling interface , 2007 .

[14]  John Doherty,et al.  Parameter estimation and uncertainty analysis for a watershed model , 2007, Environ. Model. Softw..

[15]  K. Lindenschmidt,et al.  Structural uncertainty in a river water quality modelling system , 2007 .

[16]  Jeroen P. van der Sluijs,et al.  Uncertainty and precaution in environmental management: Insights from the UPEM conference , 2007, Environ. Model. Softw..

[17]  Zhengdong Lu,et al.  Fast neural network surrogates for very high dimensional physics-based models in computational oceanography , 2007, Neural Networks.

[18]  Craig A. Aumann,et al.  A methodology for developing simulation models of complex systems , 2007 .

[19]  Alberto Montanari,et al.  What do we mean by ‘uncertainty’? The need for a consistent wording about uncertainty assessment in hydrology , 2007 .

[20]  Carlo Giupponi,et al.  Environmental decision support systems: Current issues, methods and tools , 2007, Environ. Model. Softw..

[21]  Keith Beven,et al.  Towards integrated environmental models of everywhere: uncertainty, data and modelling as a learning process , 2007 .

[22]  Jasper A Vrugt,et al.  Improved evolutionary optimization from genetically adaptive multimethod search , 2007, Proceedings of the National Academy of Sciences.

[23]  Bryan A. Tolson,et al.  Dynamically dimensioned search algorithm for computationally efficient watershed model calibration , 2007 .

[24]  Jianguo Wu,et al.  UNCERTAINTY ANALYSIS IN ECOLOGICAL STUDIES:AN OVERVIEW , 2006 .

[25]  D. Kavetski,et al.  Towards a Bayesian total error analysis of conceptual rainfall-runoff models: Characterising model error using storm-dependent parameters , 2006 .

[26]  Laura Painton Swiler,et al.  Calibration, validation, and sensitivity analysis: What's what , 2006, Reliab. Eng. Syst. Saf..

[27]  Jon C. Helton,et al.  Survey of sampling-based methods for uncertainty and sensitivity analysis , 2006, Reliab. Eng. Syst. Saf..

[28]  C. Shoemaker,et al.  Assessing the impacts of parameter uncertainty for computationally expensive groundwater models , 2006 .

[29]  Max D. Morris,et al.  Input screening: Finding the important model inputs on a budget , 2006, Reliab. Eng. Syst. Saf..

[30]  A. OHagan,et al.  Bayesian analysis of computer code outputs: A tutorial , 2006, Reliab. Eng. Syst. Saf..

[31]  K. C. Abbaspour,et al.  Calibration and uncertainty issues of a hydrological model (SWAT) applied to West Africa , 2006 .

[32]  Arnold W. Heemink,et al.  Model inversion of transient nonlinear groundwater flow models using model reduction , 2006 .

[33]  John Doherty,et al.  An advanced regularization methodology for use in watershed model calibration , 2006 .

[34]  Katri Rankinen,et al.  An application of the GLUE methodology for estimating the parameters of the INCA-N model. , 2006, The Science of the total environment.

[35]  James S. Clark,et al.  A future for models and data in environmental science. , 2006, Trends in ecology & evolution.

[36]  Anthony J. Jakeman,et al.  Ten iterative steps in development and evaluation of environmental models , 2006, Environ. Model. Softw..

[37]  J. Burnham.  Scopus database: a review , 2006, Biomedical digital libraries.

[38]  George Kuczera,et al.  Bayesian analysis of input uncertainty in hydrological modeling: 1. Theory , 2006 .

[39]  George Kuczera,et al.  Bayesian analysis of input uncertainty in hydrological modeling: 2. Application , 2006 .

[40]  John P. Norton,et al.  To what extent, and how, might uncertainty be defined? Comments engendered by "Defining uncertainty: a conceptual basis for uncertainty management in model-based decision support": Walker et al., Integrated Assessment 4: 1, 2003 , 2006 .

[41]  Panos G Georgopoulos,et al.  From a Theoretical Framework of Human Exposure and Dose Assessment to Computational System Implementation: The Modeling ENvironment for TOtal Risk Studies (MENTOR) , 2006, Journal of toxicology and environmental health. Part B, Critical reviews.

[42]  Shengtai Li,et al.  Sensitivity analysis of differential-algebraic equations and partial differential equations , 2005, Comput. Chem. Eng..

[43]  M. Bakker.  Analytic Element Modeling of Embedded Multiaquifer Domains , 2006, Ground water.

[44]  J Alex,et al.  Efficient modelling necessitates standards for model documentation and exchange. , 2006, Water science and technology : a journal of the International Association on Water Pollution Research.

[45]  A van Griensven,et al.  Methods to quantify and identify the sources of uncertainty for river basin water quality models. , 2006, Water science and technology : a journal of the International Association on Water Pollution Research.

[46]  P Reichert.  A standard interface between simulation programs and systems analysis software. , 2006, Water science and technology : a journal of the International Association on Water Pollution Research.

[47]  Markus Neteler,et al.  Free and Open Source Geospatial Tools for Environmental Modeling and Management , 2006 .

[48]  E. G. Bekele,et al.  Automatic Calibration of a Semi-Distributed Hydrologic Model Using Particle Swarm Optimization , 2005 .

[49]  Nws.  NOAA's Advanced Hydrologic Prediction Service , 2005 .

[50]  Ayalew Kassahun,et al.  Quality assurance in model based water management - review of existing practice and outline of new approaches , 2005, Environ. Model. Softw..

[51]  J. Doherty,et al.  A hybrid regularized inversion methodology for highly parameterized environmental models , 2005 .

[52]  Dongbin Xiu,et al.  High-Order Collocation Methods for Differential Equations with Random Inputs , 2005, SIAM J. Sci. Comput..

[53]  Olaf David,et al.  Developing natural resource models using the object modeling system: feasibility and challenges , 2005 .

[54]  Karl J. Castleton,et al.  Investigating uncertainty and sensitivity in integrated, multimedia environmental models: tools for FRAMES-3MRA , 2005, Environ. Model. Softw..

[55]  G. Finzi,et al.  Sensitivity analysis of ozone long term simulations to grid resolution , 2005 .

[56]  Anthony J. Jakeman,et al.  A Bayesian decision network approach for assessing the ecological impacts of salinity management , 2005, Math. Comput. Simul..

[57]  Marianthi G. Ierapetritou,et al.  A comparative assessment of efficient uncertainty analysis techniques for environmental fate and transport models: application to the FACT model , 2005 .

[58]  Arnold W. Heemink,et al.  Inverse modeling of groundwater flow using model reduction , 2005 .

[59]  Yuguo Chen.  Another look at rejection sampling through importance sampling , 2005 .

[60]  Misgana K. Muleta,et al.  Sensitivity and uncertainty analysis coupled with automatic calibration for a distributed watershed model , 2005 .

[61]  Suhrid Balakrishnan,et al.  Fast equivalent operational model of tropospheric alkane photochemistry , 2005 .

[62]  Adrian Sandu,et al.  Adjoint sensitivity analysis of regional air quality models , 2005 .

[63]  Christina W. Tsai,et al.  Evaluation of Probabilistic Point Estimate Methods in Uncertainty Analysis for Environmental Engineering Applications , 2005 .

[64]  Qingyun Duan,et al.  NOAA'S Advanced Hydrologic Prediction Service: Building Pathways for Better Science in Water Forecasting , 2005 .

[65]  Wresting Environmental Decisions From an Uncertain World , 2005 .

[66]  Holger R. Maier,et al.  A parameter-bounding approach to sensitivity assessment of large simulation models , 2005 .

[67]  James S. Clark,et al.  Why environmental scientists are becoming Bayesians , 2004 .

[68]  Ibrahim M. Khadam,et al.  Use of soft information to describe the relative uncertainty of calibration data in hydrologic models , 2004 .

[69]  John Marshall,et al.  Evaluating carbon sequestration efficiency in an ocean circulation model by adjoint sensitivity analysis , 2004 .

[70]  Fu‐Chun Wu,et al.  Second-order Monte Carlo uncertainty/variability analysis using correlated model parameters: application to salmonid embryo survival risk assessment , 2004 .

[71]  Anthony W. King,et al.  Spatial Uncertainty and Ecological Models , 2004, Ecosystems.

[72]  Henrik Madsen,et al.  Including prior information in the estimation of effective soil parameters in unsaturated zone modelling , 2004 .

[73]  J. Elsner,et al.  A Hierarchical Bayesian Approach to Seasonal Hurricane Modeling , 2004 .

[74]  Jon C. Helton,et al.  Alternative representations of epistemic uncertainty , 2004, Reliab. Eng. Syst. Saf..

[75]  R. M. Yager,et al.  Effects of Model Sensitivity and Nonlinearity on Nonlinear Regression of Ground Water Flow , 2004, Ground water.

[76]  Stefano Tarantola,et al.  Sensitivity Analysis in Practice: A Guide to Assessing Scientific Models , 2004 .

[77]  Mark E. Borsuk,et al.  A Bayesian network of eutrophication models for synthesis, prediction, and uncertainty analysis , 2004 .

[78]  Robert M. Argent,et al.  Concepts, methods and applications in environmental model integration , 2004, Environ. Model. Softw..

[79]  L. M. Berliner,et al.  Hierarchical Bayesian space-time models , 1998, Environmental and Ecological Statistics.

[80]  Jens Christian Refsgaard,et al.  Modelling guidelinesterminology and guiding principles , 2004 .

[81]  C. M. Marin,et al.  The 3MRA Risk Assessment Framework—A Flexible Approach for Performing Multimedia, Multipathway, and Multireceptor Risk Assessments Under Uncertainty , 2003 .

[82]  Igor Linkov,et al.  Model Uncertainty and Choices Made by Modelers: Lessons Learned from the International Atomic Energy Agency Model Intercomparisons † , 2003, Risk analysis : an official publication of the Society for Risk Analysis.

[83]  S. Sorooshian,et al.  Effective and efficient algorithm for multiobjective optimization of hydrologic models , 2003 .

[84]  C. Wikle.  Hierarchical Models in Environmental Science , 2003 .

[85]  David R. Anderson,et al.  Model selection and multimodel inference : a practical information-theoretic approach , 2003 .

[86]  Anthony J. Jakeman,et al.  Integrated assessment and modelling: features, principles and examples for catchment management , 2003, Environ. Model. Softw..

[87]  Helen M. Regan,et al.  Treatments of Uncertainty and Variability in Ecological Risk Assessment of Single-Species Populations , 2003 .

[88]  John Doherty,et al.  Ground Water Model Calibration Using Pilot Points and Regularization , 2003, Ground water.

[89]  Neil McIntyre,et al.  Towards reduced uncertainty in conceptual rainfall‐runoff modelling: dynamic identifiability analysis , 2003 .

[90]  Ajay K. Gupta,et al.  Latin hypercube sampling and the propagation of uncertainty in analyses of complex systems , 2002, Reliab. Eng. Syst. Saf..

[91]  R. Eganhouse,et al.  Inverse modeling of BTEX dissolution and biodegradation at the Bemidji, MN crude-oil spill site. , 2001, Journal of contaminant hydrology.

[92]  W. Walker,et al.  Defining Uncertainty: A Conceptual Basis for Uncertainty Management in Model-Based Decision Support , 2003 .

[93]  T. McGarity,et al.  Legal Aspects of the Regulatory Use of Environmental Modeling , 2003 .

[94]  Ângela Guimarães Pereira,et al.  RIVM/MNP Guidance for Uncertainty Assessment and Communication: Detailed Guidance , 2003 .

[95]  Jeffrey J. McDonnell,et al.  On the dialog between experimentalist and modeler in catchment hydrology: Use of soft data for multicriteria model calibration , 2002 .

[96]  K. Beven.  Towards a coherent philosophy for modelling the environment , 2002, Proceedings of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences.

[97]  D. Xiu,et al.  Modeling Uncertainty in Steady State Diffusion Problems via Generalized Polynomial Chaos , 2002 .

[98]  Jan Rotmans,et al.  Uncertainty in Integrated Assessment Modelling , 2002 .

[99]  G Kuczera,et al.  Rigorous uncertainty assessment in contaminant transport inverse modelling: a case study of fluoride diffusion through clay liners. , 2002, Journal of contaminant hydrology.

[100]  Enrico Zio,et al.  Solving the inverse problem of parameter estimation by genetic algorithms: the case of a groundwater contaminant transport model , 2002 .

[101]  Stephen C. Newbold.  INTEGRATED MODELING FOR WATERSHED MANAGEMENT: MULTIPLE OBJECTIVES AND SPATIAL EFFECTS1 , 2002 .

[102]  David Elizondo,et al.  Automatic versus manual model differentiation to compute sensitivities and solve non-linear inverse problems , 2002 .

[103]  Kalyanmoy Deb,et al.  A fast and elitist multiobjective genetic algorithm: NSGA-II , 2002, IEEE Trans. Evol. Comput..

[104]  D. Xiu,et al.  Stochastic Modeling of Flow-Structure Interactions Using Generalized Polynomial Chaos , 2002 .

[105]  M. Di Luzio,et al.  BETTER ASSESSMENT SCIENCE INTEGRATING POINT AND NONPOINT SOURCES (BASINS 3.0) , 2002 .

[106]  M. B. Beck.  Environmental Foresight and Models: A Manifesto , 2002 .

[107]  Ilona Bärlund,et al.  Ranking of parameters on the basis of their contribution to model uncertainty , 2001 .

[108]  Jan Rotmans,et al.  Uncertainty in Integrated Assessment Modelling: A Labyrinthic Path , 2001 .

[109]  J Rotmans,et al.  Uncertainty Management in Integrated Assessment Modeling: Towards a Pluralistic Approach , 2001, Environmental monitoring and assessment.

[110]  Pao Shan Yu,et al.  Comparison of uncertainty analysis methods for a distributed rainfall–runoff model , 2001 .

[111]  I. Sobola,et al.  Global sensitivity indices for nonlinear mathematical models and their Monte Carlo estimates , 2001 .

[112]  Cynthia Sonich-Mullin,et al.  Harmonizing the Incorporation of Uncertainty and Variability Into Risk Assessment: An International Approach , 2001 .

[113]  B. Babbitt,et al.  METHODS AND GUIDELINES FOR EFFECTIVE MODEL CALIBRATION , 2001 .

[114]  I. Sobol.  Global sensitivity indices for nonlinear mathematical models and their Monte Carlo estimates , 2001 .

[115]  Miquel Sànchez-Marrè,et al.  Environmental Decision Support Systems , 2001 .

[116]  D. Vose.  Risk Analysis: A Quantitative Guide , 2000 .

[117]  S. Sorooshian,et al.  A multistep automatic calibration scheme for river forecasting models , 2000 .

[118]  Mary C. Hill,et al.  Methods and Guidelines for Effective Model Calibration , 2000 .

[119]  Stephen R. Kraemer,et al.  Multimedia Integrated Modeling for Environmental Protection: Introduction to a Collaborative Framework , 2000 .

[120]  Fabio Gagliardi Cozman,et al.  Credal networks , 2000, Artif. Intell..

[121]  Maarten Nauta,et al.  Separation of uncertainty and variability in quantitative microbial risk assessment models. , 2000 .

[122]  Richard J. Beckman,et al.  A Comparison of Three Methods for Selecting Values of Input Variables in the Analysis of Output From a Computer Code , 2000, Technometrics.

[123]  Thorkild Hvitved-Jacobsen,et al.  Risk analysis using stochastic reliability methods applied to two cases of deterministic water quality models , 2000 .

[124]  D. Solomatine,et al.  Automatic calibration of groundwater models using global optimization techniques , 1999 .

[125]  H. Rabitz,et al.  General foundations of high‐dimensional model representations , 1999 .

[126]  H. Christopher Frey,et al.  Probabilistic Techniques in Exposure Assessment: A Handbook for Dealing with Variability and Uncertainty in Models and Inputs , 1999 .

[127]  L. Shoemaker,et al.  Better assessment science integrating point and nonpoint sources (BASINS), version 2.0. User`s manual , 1998 .

[128]  S. Isukapalli,et al.  Stochastic Response Surface Methods (SRSMs) for Uncertainty Propagation: Application to Environmental and Biological Systems , 1998, Risk analysis : an official publication of the Society for Risk Analysis.

[129]  Soroosh Sorooshian,et al.  Toward improved calibration of hydrologic models: Multiple and noncommensurable measures of information , 1998 .

[130]  Soroosh Sorooshian,et al.  Multi-objective global optimization for hydrologic models , 1998 .

[131]  David N. Lerner,et al.  How Uncertain Is Our Estimate of a Wellhead Protection Zone? , 1998 .

[132]  E. Poeter,et al.  Documentation of UCODE; a computer code for universal inverse modeling , 1998 .

[133]  A. Saltelli,et al.  Sensitivity analysis of an environmental model: an application of different analysis methods , 1997 .

[134]  M. B. Beck,et al.  On the problem of model validation for predictive exposure assessments , 1997 .

[135]  Enoch M. Dlamini,et al.  Effects of model complexity and structure, data quality, and objective functions on hydrologic modeling , 1997 .

[136]  Olli Varis,et al.  Bayesian decision analysis for environmental and resource management , 1997 .

[137]  Richard A. Davis,et al.  Introduction to time series and forecasting , 1998 .

[138]  T. Gan,et al.  Automatic Calibration of Conceptual Rainfall-Runoff Models: Optimization Algorithms, Catchment Conditions, and Model Structure , 1996 .

[139]  Clinton N Dawson,et al.  Probabilistic modeling of aquifer heterogeneity using reliability methods , 1996 .

[140]  Christian Bischof,et al.  Adifor 2.0: automatic differentiation of Fortran 77 programs , 1996 .

[141]  S. L. Markstrom,et al.  The modular modeling system (MMS) — The physical process modeling component of a database-centered decision support system for water and power management , 1996 .

[142]  P. Young,et al.  Simplicity out of complexity in environmental modelling: Occam's razor revisited. , 1996 .

[143]  Andreas Griewank,et al.  Algorithm 755: ADOL-C: a package for the automatic differentiation of algorithms written in C/C++ , 1996, TOMS.

[144]  J. Norton.  Roles for deterministic bounding in environmental modelling , 1996 .

[145]  David Andrew Barry,et al.  Assessing uncertainty in subsurface solute transport: efficient first-order reliability methods , 1995 .

[146]  F. O. Hoffman,et al.  Propagation of uncertainty in risk assessments: the need to distinguish between uncertainty due to lack of knowledge and uncertainty due to variability. , 1994, Risk analysis : an official publication of the Society for Risk Analysis.

[147]  D. Hamby.  A review of techniques for parameter sensitivity analysis of environmental models , 1994, Environmental monitoring and assessment.

[148]  A. Jakeman,et al.  How much complexity is warranted in a rainfall‐runoff model? , 1993 .

[149]  Serafín Moral,et al.  An axiomatic framework for propagating uncertainty in directed acyclic networks , 1993, Int. J. Approx. Reason..

[150]  Keith Beven,et al.  The future of distributed models: model calibration and uncertainty prediction. , 1992 .

[151]  S. Sorooshian,et al.  Effective and efficient global optimization for conceptual rainfall‐runoff models , 1992 .

[152]  John D. Bredehoeft,et al.  Ground-water models cannot be validated , 1992 .

[153]  J. Freidman,et al.  Multivariate adaptive regression splines , 1991 .

[154]  Jerome R. Ravetz,et al.  Uncertainty and Quality in Science for Policy , 1990 .

[155]  Max Henrion,et al.  Uncertainty: A Guide to Dealing with Uncertainty in Quantitative Risk and Policy Analysis , 1990 .

[156]  Hannes Werthner,et al.  Environmental decision support systems , 1989 .

[157]  Petre Stoica,et al.  Decentralized Control , 2018, The Control Systems Handbook.

[158]  M. B. Beck,et al.  Water quality modeling: A review of the analysis of uncertainty , 1987 .

[159]  George Z. Gertner,et al.  Approximating Precision in Simulation Projections: An Efficient Alternative to Monte Carlo Methods , 1987, Forest Science.

[160]  J. P. Norton,et al.  Identification and application of bounded-parameter models , 1985, Autom..

[161]  Lennart Ljung,et al.  System Identification: Theory for the User , 1987 .

[162]  R. Iman,et al.  A distribution-free approach to inducing rank correlation among input variables , 1982 .

[163]  D. C. Cox,et al.  Methods for Uncertainty Analysis: A Comparative Survey , 1981 .

[164]  G. Hornberger,et al.  Approach to the preliminary analysis of environmental systems , 1981 .

[165]  R. Spear.  Eutrophication in peel inlet—II. Identification of critical uncertainties via generalized sensitivity analysis , 1980 .

[166]  K.,et al.  Nonlinear sensitivity analysis of multiparameter model systems , 1977 .

[167]  J. Nash,et al.  River flow forecasting through conceptual models part I — A discussion of principles☆ , 1970 .

[168]  D. Marquardt.  An Algorithm for Least-Squares Estimation of Nonlinear Parameters , 1963 .

[169]  Kenneth Levenberg.  A METHOD FOR THE SOLUTION OF CERTAIN NON – LINEAR PROBLEMS IN LEAST SQUARES , 1944 .