Modifying Goodness-of-Fit Indicators to Incorporate Both Measurement and Model Uncertainty in Model Calibration and Validation

Because of numerous practical implications of uncertainty in measured data and model predictions, improved techniques are needed to analyze and understand uncertainty and incorporate it into hydrologic and water quality evaluations. In the present study, a correction factor was developed to incorporate measurement uncertainty and model uncertainty in evaluations of model goodness-of-fit (predictive ability). The correction factor, which was developed for pairwise comparisons of measured and predicted values, modifies the typical error term calculation to consider both sources of uncertainty. The correction factor was applied with common distributions and levels of uncertainty (represented by coefficients of variation ranging from 0.026 to 0.256) for each measured value and each predicted value from five example data sets. The modifications resulted in inconsequential changes in goodness-of-fit conclusions for example data sets with very good and poor model simulations, which is both logical and appropriate because very good model performance should not improve greatly and poor model performance should not become satisfactory when uncertainty is considered. In contrast, incorporating uncertainty in example data sets with initially moderate goodness-of-fit resulted in important improvements in indicator values and in model performance ratings. A model evaluation matrix was developed to present appropriate model performance conclusions, considering both model accuracy and precision, based on various levels of measurement and model uncertainty. In cases with highly uncertain calibration/validation data, definitive "good" fit conclusions are cautioned against even with "good" indicator values because of the uncertain standard of comparison; however, in these cases, poor model accuracy can be confidently concluded from "unsatisfactory" indicator values. In contrast, model accuracy can be confidently concluded from goodness-of-fit indicator values in cases with low measurement uncertainty. It is hoped that the modified goodness-of-fit indicators and the model evaluation matrix contribute to improved goodness-of-fit conclusions and to more complete assessments of model performance.

[1]  Keith Beven,et al.  Equifinality, data assimilation, and uncertainty estimation in mechanistic modelling of complex environmental systems using the GLUE methodology , 2001 .

[2]  C. T. Haan,et al.  Uncertainty analysis using corrected first‐order approximation method , 2001 .

[3]  C. T. Haan,et al.  Statistical Procedure for Evaluating Hydrologic/Water Quality Models , 1995 .

[4]  John R. Williams,et al.  EPIC-erosion/productivity impact calculator: 1. Model documentation. , 1990 .

[5]  C. Willmott.  ON THE VALIDATION OF MODELS , 1981 .

[6]  Jeffrey G. Arnold,et al.  Model Evaluation Guidelines for Systematic Quantification of Accuracy in Watershed Simulations , 2007 .

[7]  F. Pappenberger,et al.  Ignorance is bliss: Or seven reasons not to use uncertainty analysis , 2006 .

[8]  Qian Hong,et al.  Parameter uncertainty analysis of the non-point source pollution in the Daning River watershed of the Three Gorges Reservoir Region, China. , 2008, The Science of the total environment.

[9]  K. Abbaspour,et al.  Modelling hydrology and water quality in the pre-alpine/alpine Thur watershed using SWAT , 2007 .

[10]  S. Sorooshian,et al.  Effective and efficient global optimization for conceptual rainfall‐runoff models , 1992 .

[11]  R. W. Skaggs,et al.  EFFECT OF PARAMETER UNCERTAINTY ON DRAINMOD PREDICTIONS: II. NITROGEN LOSS , 2003 .

[12]  Christine A. Shoemaker,et al.  Efficient prediction uncertainty approximation in the calibration of environmental simulation models , 2008 .

[13]  S. Jain,et al.  Fitting of Hydrologic Models: A Close Look at the Nash–Sutcliffe Index , 2008 .

[14]  J. Nash,et al.  River flow forecasting through conceptual models part I — A discussion of principles☆ , 1970 .

[15]  D. R. Smith,et al.  Estimating storm discharge and water quality data uncertainty: A software tool for monitoring and modeling applications , 2009, Environ. Model. Softw..

[16]  John R. Williams,et al.  LARGE AREA HYDROLOGIC MODELING AND ASSESSMENT PART I: MODEL DEVELOPMENT 1 , 1998 .

[17]  Leo R Beard,et al.  Statistical Methods in Hydrology , 1962 .

[18]  C. T. Haan.  Parametric Uncertainty in Hydrologic Modeling , 1989 .

[19]  N. Metropolis,et al.  Equation of State Calculations by Fast Computing Machines , 1953, Resonance.

[20]  Kenneth H. Reckhow.  On the Need for Uncertainty Assessment in TMDL Modeling and Implementation , 2003 .

[21]  M. B. Beck,et al.  Water quality modeling: A review of the analysis of uncertainty , 1987 .

[22]  Mazdak Arabi,et al.  A Hydrologic/Water Quality Model Applicati1 1 , 2007 .

[23]  Patrice M. Pelletier,et al.  Uncertainties in the single determination of river discharge: a literature review , 1988 .

[24]  R. W. Skaggs,et al.  EFFECT OF PARAMETER UNCERTAINTY ON DRAINMOD PREDICTIONS: I. HYDROLOGY AND YIELD , 2003 .

[25]  Misgana K. Muleta,et al.  Sensitivity and uncertainty analysis coupled with automatic calibration for a distributed watershed model , 2005 .

[26]  Andrew N. Sharpley,et al.  EPIC, Erosion/Productivity Impact Calculator , 1990 .

[27]  M. D. McKay,et al.  A comparison of three methods for selecting values of input variables in the analysis of output from a computer code , 2000 .

[28]  Jing Yang,et al.  Comparing uncertainty analysis techniques for a SWAT application to the Chaohe Basin in China , 2008 .

[29]  David B. Dunson,et al.  Bayesian Data Analysis , 2010 .

[30]  Keith Beven,et al.  Changing ideas in hydrology — The case of physically-based models , 1989 .

[31]  R. McCuen,et al.  Evaluation of the Nash-Sutcliffe Efficiency Index , 2006 .

[32]  Marty D. Matlock,et al.  A Watershed-Level Ecological Risk Assessment Methodology , 1996 .

[33]  C. T. Haan,et al.  Effect of Parameter Distributions on Uncertainty Analysis of Hydrologic Models , 1998 .

[34]  Niels C. Lind,et al.  Methods of structural safety , 2006 .

[35]  B. Chessman,et al.  Effects of water sample preservation and storage on nitrogen and phosphorus determinations: implications for the use of automated sampling equipment , 1998 .

[36]  A van Griensven,et al.  Methods to quantify and identify the sources of uncertainty for river basin water quality models. , 2006, Water science and technology : a journal of the International Association on Water Pollution Research.

[37]  Jeffrey G. Arnold,et al.  HYDROLOGIC SIMULATION ON AGRICULTURAL WATERSHEDS: CHOOSING BETWEEN TWO MODELS , 2003 .

[38]  George Kuczera,et al.  Monte Carlo assessment of parameter uncertainty in conceptual catchment models: the Metropolis algorithm , 1998 .

[39]  R. Daren Harmel,et al.  Consideration of measurement uncertainty in the evaluation of goodness-of-fit in hydrologic and water quality modeling , 2007 .

[40]  Indrajeet Chaubey,et al.  UNCERTAINTY IN TMDL MODELS , 2006 .

[41]  Jeffrey G. Arnold,et al.  CUMULATIVE UNCERTAINTY IN MEASURED STREAMFLOW AND WATER QUALITY DATA FOR SMALL WATERSHEDS , 2006 .

[42]  Richard J. Beckman,et al.  A Comparison of Three Methods for Selecting Values of Input Variables in the Analysis of Output From a Computer Code , 2000, Technometrics.

[43]  K. Loague,et al.  Statistical and graphical methods for evaluating solute transport models: Overview and application , 1991 .

[44]  J. Arnold,et al.  VALIDATION OF THE SWAT MODEL ON A LARGE RWER BASIN WITH POINT AND NONPOINT SOURCES 1 , 2001 .

[45]  Wesley W. Wallender,et al.  Evaluation of Modeling Tools For TMDL Development And Implementation , 2007 .

[46]  Guillermo J. Vicens,et al.  A Bayesian framework for the use of regional information in hydrology , 1975 .

[47]  K. Beven.  On undermining the science? , 2006 .

[48]  Cole H Green,et al.  HYDROLOGIC EVALUATION OF THE SOIL AND WATER ASSESSMENT TOOL FOR A LARGE TILE-DRAINED WATERSHED IN IOWA , 2006 .

[49]  Indrajeet Chaubey,et al.  Spatial Distributions and Stochastic Parameter Influences on SWAT Flow and Sediment Predictions , 2008 .

[50]  D. Legates,et al.  Evaluating the use of “goodness‐of‐fit” Measures in hydrologic and hydroclimatic model validation , 1999 .