Evaluating, interpreting, and communicating performance of hydrologic/water quality models considering intended use: A review and recommendations

Abstract Previous publications have outlined recommended practices for hydrologic and water quality (H/WQ) modeling, but limited guidance has been published on how to consider the project's purpose or model's intended use, especially for the final stage of modeling applications – namely evaluation, interpretation, and communication of model results. Such guidance is needed to more effectively evaluate and interpret model performance and more accurately communicate that performance to decision-makers and other modeling stakeholders. Thus, we formulated a methodology for evaluation, interpretation, and communication of H/WQ model results. The recommended methodology focuses on interpretation and communication of results, not on model development or initial calibration and validation, and as such it applies to the modeling process following initial calibration. The methodology recommends the following steps: 1) evaluate initial model performance; 2) evaluate outliers and extremes in observed values and bias in predicted values; 3) estimate uncertainty in observed data and predicted values; 4) re-evaluate model performance considering accuracy, precision, and hypothesis testing; 5) interpret model results considering intended use; and 6) communicate model performance. A flowchart and tables were developed to guide model interpretation, refinement, and proper application considering intended model uses (i.e., Exploratory, Planning, and Regulatory/Legal). The methodology was designed to enhance application of H/WQ models through conscientious evaluation, interpretation, and communication of model performance to decision-makers and other stakeholders; it is not meant to be a definitive standard or a required protocol, but together with recent recommendations and published best practices serve as guidelines for enhanced model application emphasizing the importance of the model's intended use.

[1]  Jeffrey G. Arnold,et al.  Model Evaluation Guidelines for Systematic Quantification of Accuracy in Watershed Simulations , 2007 .

[2]  F. Pappenberger,et al.  Ignorance is bliss: Or seven reasons not to use uncertainty analysis , 2006 .

[3]  C. T. Haan,et al.  Statistical Methods In Hydrology , 1977 .

[4]  Kati W. Migliaccio,et al.  Modifying Goodness-of-Fit Indicators to Incorporate Both Measurement and Model Uncertainty in Model Calibration and Validation , 2010 .

[5]  C. T. Haan.  Parametric Uncertainty in Hydrologic Modeling , 1989 .

[6]  H. Riedwyl.  Goodness of Fit , 1967 .

[7]  Boris Kompare,et al.  Environmental Modelling & Software , 2014 .

[8]  Robert E. Davis,et al.  Statistics for the evaluation and comparison of models , 1985 .

[9]  D. Kavetski,et al.  Confronting Input Uncertainty in Environmental Modelling , 2013 .

[10]  Robert H. Montgomery,et al.  Uncertainty in Water Quality Data , 1986 .

[11]  D. R. Smith,et al.  Estimating storm discharge and water quality data uncertainty: A software tool for monitoring and modeling applications , 2009, Environ. Model. Softw..

[12]  S. Jain,et al.  Fitting of Hydrologic Models: A Close Look at the Nash–Sutcliffe Index , 2008 .

[13]  K. Taylor.  Summarizing multiple aspects of model performance in a single diagram , 2001 .

[14]  Patrick Willems,et al.  A non-parametric data-based approach for probabilistic flood forecasting in support of uncertainty communication , 2012, Environ. Model. Softw..

[15]  R. McCuen,et al.  Evaluation of the Nash-Sutcliffe Efficiency Index , 2006 .

[16]  Michael H. Ramsey,et al.  Sampling as a source of measurement uncertainty: techniques for quantification and comparison with analytical sources , 1998 .

[17]  Drainage Division,et al.  Criteria for Evaluation of Watershed Models , 1993 .

[18]  R. Daren Harmel,et al.  Consideration of measurement uncertainty in the evaluation of goodness-of-fit in hydrologic and water quality modeling , 2007 .

[19]  S. Lorenz,et al.  The communication of physical science uncertainty in European National Adaptation Strategies , 2013, Climatic Change.

[20]  Hoshin Vijai Gupta,et al.  Do Nash values have value? , 2007 .

[21]  Kenneth H. Reckhow.  On the Need for Uncertainty Assessment in TMDL Modeling and Implementation , 2003 .

[22]  Keith Beven,et al.  Equifinality, data assimilation, and uncertainty estimation in mechanistic modelling of complex environmental systems using the GLUE methodology , 2001 .

[23]  Beat Kleiner,et al.  Graphical Methods for Data Analysis , 1983 .

[24]  J. Nash,et al.  A criterion of efficiency for rainfall-runoff models , 1978 .

[25]  Joseph H. A. Guillaume,et al.  Characterising performance of environmental models , 2013, Environ. Model. Softw..

[26]  A. Patt,et al.  Communicating uncertainty: lessons learned and suggestions for climate change assessment , 2005 .

[27]  M. B. Beck,et al.  Water quality modeling: A review of the analysis of uncertainty , 1987 .

[28]  Glyn Elwyn,et al.  Communicating uncertainty can lead to less decision satisfaction: a necessary cost of involving patients in shared decision making? , 2011, Health expectations : an international journal of public participation in health care and health policy.

[29]  Mazdak Arabi,et al.  A Hydrologic/Water Quality Model Applicati1 1 , 2007 .

[30]  Gabriele Freni,et al.  Validation of hydrological models: conceptual basis, methodological approaches and a proposal for a code of practice , 2010 .

[31]  K. Loague,et al.  Statistical and graphical methods for evaluating solute transport models: Overview and application , 1991 .

[32]  Volker Grimm,et al.  Merging validation and evaluation of ecological models to ‘evaludation’: A review of terminology and a practical approach , 2014 .

[33]  Paul K J Han,et al.  Conceptual, Methodological, and Ethical Problems in Communicating Uncertainty in Clinical Evidence , 2013, Medical care research and review : MCRR.

[34]  Anthony J. Jakeman,et al.  Ten iterative steps in development and evaluation of environmental models , 2006, Environ. Model. Softw..

[35]  R. Muñoz‐Carpena,et al.  Performance evaluation of hydrological models: Statistical significance for reducing subjectivity in goodness-of-fit assessments , 2013 .

[36]  D. Legates,et al.  A refined index of model performance: a rejoinder , 2013 .

[37]  Indrajeet Chaubey,et al.  Uncertainty in TMDL Models , 2006 .

[38]  Indrajeet Chaubey,et al.  Simulated watershed scale impacts of corn stover removal for biofuel on hydrology and water quality , 2012 .

[39]  D. C. Black,et al.  Towards best practice implementation and application of models for analysis of water resources management scenarios , 2014, Environ. Model. Softw..

[40]  S. Broomell,et al.  Effective communication of uncertainty in the IPCC reports , 2012, Climatic Change.

[41]  Guillermo J. Vicens,et al.  A Bayesian framework for the use of regional information in hydrology , 1975 .

[42]  Qi Zhang,et al.  Parameter and modeling uncertainty simulated by GLUE and a formal Bayesian method for a conceptual hydrological model , 2010 .

[43]  C. Willmott,et al.  A refined index of model performance , 2012 .

[44]  P. E. O'connell,et al.  River flow forecasting through conceptual models part III - The Ray catchment at Grendon Underwood , 1970 .

[45]  Wesley W. Wallender,et al.  EVALUATION OF MODELING TOOLS FOR TMDL DEVELOPMENT AND IMPLEMENTATION , 2006 .

[46]  K. Beven.  On undermining the science? , 2006 .

[47]  D. Legates,et al.  Evaluating the use of “goodness‐of‐fit” Measures in hydrologic and hydroclimatic model validation , 1999 .

[48]  Keith Beven,et al.  Changing ideas in hydrology — The case of physically-based models , 1989 .

[49]  Jeffrey G. Arnold,et al.  CUMULATIVE UNCERTAINTY IN MEASURED STREAMFLOW AND WATER QUALITY DATA FOR SMALL WATERSHEDS , 2006 .

[50]  John M. Chambers,et al.  Graphical Methods for Data Analysis , 1983 .

[51]  Ayalew Kassahun,et al.  Quality assurance in model based water management - review of existing practice and outline of new approaches , 2005, Environ. Model. Softw..

[52]  M. Katz.  Validation of models , 2006 .

[53]  J. Nash,et al.  River flow forecasting through conceptual models part I — A discussion of principles☆ , 1970 .

[54]  J. A. Wardekker,et al.  Uncertainty Communication : Issues and good practice , 2007 .

[55]  Kyle R. Douglas-Mankin,et al.  Hydrologic and Water Quality Models: Use, Calibration, and Validation , 2012 .

[56]  C. T. Haan,et al.  Statistical Procedure for Evaluating Hydrologic/Water Quality Models , 1995 .

[57]  Soroosh Sorooshian,et al.  A framework for development and application of hydrological models , 2001, Hydrology and Earth System Sciences.

[58]  Timothy A. Cohn,et al.  Load Estimator (LOADEST): A FORTRAN Program for Estimating Constituent Loads in Streams and Rivers , 2004 .