Evaluating model structure adequacy: The case of the Maggia Valley groundwater system, southern Switzerland

[1] Model adequacy is evaluated with alternative models rated using model selection criteria (AICc, BIC, and KIC) and three other statistics. Model selection criteria are tested with cross-validation experiments and insights for using alternative models to evaluate model structural adequacy are provided. The study is conducted using the computer codes UCODE_2005 and MMA (MultiModel Analysis). One recharge alternative is simulated using the TOPKAPI hydrological model. The predictions evaluated include eight heads and three flows located where ecological consequences and model precision are of concern. Cross-validation is used to obtain measures of prediction accuracy. Sixty-four models were designed deterministically and differ in representation of river, recharge, bedrock topography, and hydraulic conductivity. Results include: (1) What may seem like inconsequential choices in model construction may be important to predictions. Analysis of predictions from alternative models is advised. (2) None of the model selection criteria consistently identified models with more accurate predictions. This is a disturbing result that suggests to reconsider the utility of model selection criteria, and/or the cross-validation measures used in this work to measure model accuracy. (3) KIC displayed poor performance for the present regression problems; theoretical considerations suggest that difficulties are associated with wide variations in the sensitivity term of KIC resulting from the models being nonlinear and the problems being ill-posed due to parameter correlations and insensitivity. The other criteria performed somewhat better, and similarly to each other. (4) Quantities with high leverage are more difficult to predict. The results are expected to be generally applicable to models of environmental systems.

[1]  H. Akaike.  A new look at the statistical model identification , 1974 .

[2]  Heidi Christiansen Barlebo,et al.  Concentration data and dimensionality in groundwater models: evaluation using inverse modelling , 1998 .

[3]  D. Sweetkind,et al.  Death Valley regional groundwater flow system, Nevada and California : hydrogeologic framework and transient groundwater flow model , 2010 .

[4]  Ming Ye,et al.  Dependence of Bayesian Model Selection Criteria and Fisher Information Matrix on Sample Size , 2011 .

[5]  B. Efron.  The jackknife, the bootstrap, and other resampling plans , 1987 .

[6]  John Doherty,et al.  Use of paired simple and complex models to reduce predictive bias and quantify uncertainty , 2011 .

[7]  S. P. Neuman,et al.  On model selection criteria in multimodel analysis , 2007 .

[8]  Heidi Christiansen Barlebo,et al.  Investigating the Macrodispersion Experiment (MADE) site in Columbus, Mississippi, using a three‐dimensional inverse flow and transport model , 2004 .

[9]  N. Draper,et al.  Applied Regression Analysis , 1967 .

[10]  Ming Ye,et al.  Maximum likelihood Bayesian averaging of spatial variability models in unsaturated fractured tuff , 2003 .

[11]  George Kuczera,et al.  Toward a reliable decomposition of predictive uncertainty in hydrological modeling: Characterizing rainfall errors using conditional simulation , 2011 .

[12]  Mary C. Hill,et al.  Comment on ``Two statistics for evaluating parameter identifiability and error reduction'' by John Doherty and Randall J. Hunt , 2010 .

[13]  Jens Christian Refsgaard,et al.  Review of strategies for handling geological uncertainty in groundwater flow and transport modeling , 2012 .

[14]  N. Draper,et al.  Applied Regression Analysis: Draper/Applied Regression Analysis , 1998 .

[15]  J. Deleeuw,et al.  Introduction to Akaike (1973) Information Theory and an Extension of the Maximum Likelihood Principle , 1992 .

[16]  Zucchini,et al.  An Introduction to Model Selection. , 2000, Journal of mathematical psychology.

[17]  M. C. Hill,et al.  Unrealistic parameter estimates in inverse modelling: A problem or a benefit for model calibration? , 1996 .

[18]  S. P. Neuman,et al.  Estimation of Aquifer Parameters Under Transient and Steady State Conditions: 1. Maximum Likelihood Method Incorporating Prior Information , 1986 .

[19]  C. Tiedeman,et al.  Effective Groundwater Model Calibration , 2007 .

[20]  Ming Ye,et al.  Analysis of regression confidence intervals and Bayesian credible intervals for uncertainty quantification , 2012 .

[21]  J. Kirchner.  Getting the right answers for the right reasons: Linking measurements, analyses, and models to advance the science of hydrology , 2006 .

[22]  George Kuczera,et al.  Understanding predictive uncertainty in hydrologic modeling: The challenge of identifying input and structural errors , 2010 .

[23]  Arlen W. Harbaugh,et al.  MODFLOW-2000, The U.S. Geological Survey Modular Ground-Water Model - User Guide to Modularization Concepts and the Ground-Water Flow Process , 2000 .

[24]  S. P. Neuman,et al.  Inverse stochastic moment analysis of steady state flow in randomly heterogeneous media , 2006 .

[25]  Jens Christian Refsgaard,et al.  Assessment of hydrological model predictive ability given multiple conceptual geological models , 2012 .

[26]  L Foglia,et al.  Testing Alternative Ground Water Models Using Cross‐Validation and Other Methods , 2007, Ground water.

[27]  Ming Ye,et al.  A Model‐Averaging Method for Assessing Groundwater Conceptual Model Uncertainty , 2010, Ground water.

[28]  Claire R. Tiedeman,et al.  OPR-PPR, a Computer Program for Assessing Data Importance to Model Predictions Using Linear Statistics , 2014 .

[29]  Ming Ye,et al.  Towards a comprehensive assessment of model structural adequacy , 2012 .

[30]  Martyn P. Clark,et al.  Framework for Understanding Structural Errors (FUSE): A modular framework to diagnose differences between hydrological models , 2008 .

[31]  Steffen Mehl,et al.  Grid-size dependence of Cauchy boundary conditions used to simulate stream–aquifer interactions , 2010 .

[32]  David Anderson,et al.  Multimodel Ranking and Inference in Ground Water Modeling , 2004, Ground water.

[33]  Dmitri Kavetski,et al.  Impact of temporal data resolution on parameter inference and model identification in conceptual hydrological modeling: Insights from an experimental catchment , 2010 .

[34]  John Doherty,et al.  Two statistics for evaluating parameter identifiability and error reduction , 2009 .

[35]  Ming Ye,et al.  Comment on “Inverse groundwater modeling for hydraulic conductivity estimation using Bayesian model averaging and variance window” by Frank T.‐C. Tsai and Xiaobao Li , 2010 .

[36]  Saltelli Andrea,et al.  Global Sensitivity Analysis: The Primer , 2008 .

[37]  D. M. Allen.  Mean Square Error of Prediction as a Criterion for Selecting Variables , 1971 .

[38]  D. Krabbenhoft,et al.  Obtaining parsimonious hydraulic conductivity fields using head and transport observations: A Bayesian geostatistical parameter estimation approach , 2009 .

[39]  Mario L. V. Martina,et al.  Flood forecasting using a fully distributed model: application of the TOPKAPI model to the Upper Xixian Catchment , 2005 .

[40]  S. P. Neuman,et al.  Maximum likelihood Bayesian averaging of uncertain model predictions , 2003 .

[41]  Frank T.-C. Tsai,et al.  Inverse groundwater modeling for hydraulic conductivity estimation using Bayesian model averaging and variance window , 2008 .

[42]  Ming Ye,et al.  Combined Estimation of Hydrogeologic Conceptual Model, Parameter, and Scenario Uncertainty with Application to Uranium Transport at the Hanford Site 300 Area , 2006 .

[43]  David R. Anderson,et al.  Model selection and multimodel inference : a practical information-theoretic approach , 2003 .

[44]  Mary C. Hill,et al.  UCODE_2005 and six other computer codes for universal sensitivity analysis, calibration, and uncertainty evaluation constructed using the JUPITER API , 2006 .

[45]  Mary C Hill,et al.  The Practical Use of Simplicity in Developing Ground Water Models , 2006, Ground water.

[46]  S. P. Neuman,et al.  Sensitivity analysis and assessment of prior model probabilities in MLBMA with application to unsaturated fractured tuff , 2005 .

[47]  Mary C. Hill,et al.  Sensitivity analysis, calibration, and testing of a distributed hydrological model using error‐based weighting and one objective function , 2009 .

[48]  John Doherty,et al.  A short exploration of structural noise , 2010 .

[49]  Frank T.-C. Tsai,et al.  Reply to comment by Ming Ye et al. on “Inverse groundwater modeling for hydraulic conductivity estimation using Bayesian model averaging and variance window” , 2010 .

[50]  Frank T.-C. Tsai,et al.  Bayesian model averaging for groundwater head prediction and uncertainty analysis using multimodel and multimethod , 2009 .

[51]  Dmitri Kavetski,et al.  Pursuing the method of multiple working hypotheses for hydrological modeling , 2011 .

[52]  Ming Ye,et al.  MMA: A Computer Code for Multimodel Analysis , 2010 .

[53]  Edward R. Banta,et al.  A new streamflow-routing (SFR1) package to simulate stream-aquifer interaction with MODFLOW-2000 , 2004 .

[54]  Jeroen P. van der Sluijs,et al.  A framework for dealing with uncertainty due to model structure error , 2004 .

[55]  R. Liedl,et al.  Complexity versus simplicity: an example of groundwater model ranking with the Akaike Information Criterion , 2012 .

[56]  C. Faunt,et al.  Groundwater availability of the Central Valley Aquifer, California , 2009 .

[57]  Sean A. McKenna,et al.  Reducing Uncertainty Associated with Ground‐Water Flow and Transport Predictions , 1995 .

[58]  Mary C. Hill,et al.  Predictive modeling of flow and transport in a two‐dimensional intermediate‐scale, heterogeneous porous medium , 2001 .

[59]  Rangasami L. Kashyap,et al.  Optimal Choice of AR and MA Parts in Autoregressive Moving Average Models , 1982, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[60]  David R. Anderson,et al.  Multimodel Inference , 2004 .

[61]  A. Alcolea,et al.  Exact sensitivity matrix and influence of the number of pilot points in the geostatistical inversion of moment equations of groundwater flow , 2010 .

[62]  Srikanta Mishra,et al.  Model Averaging Techniques for Quantifying Conceptual Model Uncertainty , 2010, Ground water.

[63]  Mary C. Hill,et al.  Two-dimensional advective transport in ground-water flow parameter estimation , 1996 .