Effects of error covariance structure on estimation of model averaging weights and predictive performance

[1] When conducting model averaging for assessing groundwater conceptual model uncertainty, the averaging weights are often evaluated using model selection criteria such as AIC, AICc, BIC, and KIC (Akaike Information Criterion, Corrected Akaike Information Criterion, Bayesian Information Criterion, and Kashyap Information Criterion, respectively). However, this method often leads to an unrealistic situation in which the best model receives overwhelmingly large averaging weight (close to 100%), which cannot be justified by available data and knowledge. It was found in this study that this problem was caused by using the covariance matrix, Cɛ, of measurement errors for estimating the negative log likelihood function common to all the model selection criteria. This problem can be resolved by using the covariance matrix, Cek, of total errors (including model errors and measurement errors) to account for the correlation between the total errors. An iterative two-stage method was developed in the context of maximum likelihood inverse modeling to iteratively infer the unknown Cek from the residuals during model calibration. The inferred Cek was then used in the evaluation of model selection criteria and model averaging weights. While this method was limited to serial data using time series techniques in this study, it can be extended to spatial data using geostatistical techniques. The method was first evaluated in a synthetic study and then applied to an experimental study, in which alternative surface complexation models were developed to simulate column experiments of uranium reactive transport. It was found that the total errors of the alternative models were temporally correlated due to the model errors. The iterative two-stage method using Cek resolved the problem that the best model receives 100% model averaging weight, and the resulting model averaging weights were supported by the calibration results and physical understanding of the alternative models. Using Cek obtained from the iterative two-stage method also improved predictive performance of the individual models and model averaging in both synthetic and experimental studies.

[1]  Daniel M. Tartakovsky,et al.  Assessment and management of risk in subsurface hydrology: A review and perspective , 2013 .

[2]  Ming Ye,et al.  Maximum likelihood Bayesian averaging of spatial variability models in unsaturated fractured tuff , 2003 .

[3]  P. Mahalanobis.  On the generalized distance in statistics , 1936 .

[4]  Keith Beven,et al.  A manifesto for the equifinality thesis , 2006 .

[5]  G. Schwarz.  Estimating the Dimension of a Model , 1978 .

[6]  N. Hjort,et al.  Frequentist Model Average Estimators , 2003 .

[7]  R. Guillaumont,et al.  Update on the chemical thermodynamics of uranium, neptunium, plutonium, americium and technetium , 2003 .

[8]  Alain Dassargues,et al.  Sensitivity analysis of prior model probabilities and the value of prior knowledge in the assessment of conceptual model uncertainty in groundwater modelling , 2009 .

[9]  Srikanta Mishra,et al.  Model Averaging Techniques for Quantifying Conceptual Model Uncertainty , 2010, Ground water.

[10]  David Anderson,et al.  Multimodel Ranking and Inference in Ground Water Modeling , 2004, Ground water.

[11]  S. P. Neuman,et al.  Multimodel Bayesian analysis of data-worth applied to unsaturated fractured tuffs , 2012 .

[12]  Y. Rubin,et al.  A hypothesis‐driven approach to optimize field campaigns , 2012 .

[13]  John Doherty,et al.  A short exploration of structural noise , 2010 .

[14]  Bruce A. Robinson,et al.  Treatment of uncertainty using ensemble methods: Comparison of sequential data assimilation and Bayesian model averaging , 2007 .

[15]  Chunmiao Zheng,et al.  MMA: A Computer Code for Multimodel Analysis , 2010 .

[16]  J. Vrugt,et al.  A formal likelihood function for parameter and predictive inference of hydrologic models with correlated, heteroscedastic, and non‐Gaussian errors , 2010 .

[17]  Chris Chatfield,et al.  The Analysis of Time Series , 1990 .

[18]  S. P. Neuman,et al.  Estimation of Aquifer Parameters Under Transient and Steady State Conditions: 1. Maximum Likelihood Method Incorporating Prior Information , 1986 .

[19]  Stefan Finsterle,et al.  Error handling strategies in multiphase inverse modeling , 2011, Comput. Geosci..

[20]  Ming Ye,et al.  Quantification of model uncertainty in environmental modeling , 2010 .

[21]  Adrian E. Raftery,et al.  Bayesian model averaging: a tutorial (with comments by M. Clyde, David Draper and E. I. George, and a rejoinder by the authors , 1999 .

[22]  L Foglia,et al.  Testing Alternative Ground Water Models Using Cross‐Validation and Other Methods , 2007, Ground water.

[23]  Ming Ye,et al.  A Model‐Averaging Method for Assessing Groundwater Conceptual Model Uncertainty , 2010, Ground water.

[24]  Ming Ye,et al.  Combined Estimation of Hydrogeologic Conceptual Model, Parameter, and Scenario Uncertainty with Application to Uranium Transport at the Hanford Site 300 Area , 2006 .

[25]  C. Bishop,et al.  Climate model dependence and the replicate Earth paradigm , 2013, Climate Dynamics.

[26]  Johan Alexander Huisman,et al.  Bayesian model averaging using particle filtering and Gaussian mixture modeling: Theory, concepts, and simulation experiments , 2012 .

[27]  R. Lyman Ott.,et al.  An introduction to statistical methods and data analysis , 1977 .

[28]  S. P. Neuman,et al.  On model selection criteria in multimodel analysis , 2007 .

[29]  Ashish Sharma,et al.  Hydrological model selection: A Bayesian alternative , 2005 .

[30]  S. Sorooshian,et al.  Stochastic parameter estimation procedures for hydrologie rainfall‐runoff models: Correlated and heteroscedastic error cases , 1980 .

[31]  Ming Ye,et al.  Use of Numerical Groundwater Modeling to Evaluate Uncertainty in Conceptual Models of Recharge and Hydrostratigraphy , 2007, 2007 IEEE International Symposium on Technology and Society.

[32]  S. Weisberg,et al.  Residuals and Influence in Regression , 1982 .

[33]  W. Yeh,et al.  Parameter Identification of Groundwater Aquifer Models: A Generalized Least Squares Approach , 1984 .

[34]  Ming Ye,et al.  Towards a comprehensive assessment of model structural adequacy , 2012 .

[35]  Matthias Kohler,et al.  Experimental Investigation and Modeling of Uranium (VI) Transport Under Variable Chemical Conditions , 1996 .

[36]  Clifford H. Thurber,et al.  Parameter estimation and inverse problems , 2005 .

[37]  S. P. Neuman,et al.  Role of model selection criteria in geostatistical inverse estimation of statistical data‐ and model‐parameters , 2011 .

[38]  D. Madigan,et al.  Bayesian Model Averaging for Linear Regression Models , 1997 .

[39]  L. Shawn Matott,et al.  Evaluating uncertainty in integrated environmental models: A review of concepts and tools , 2009 .

[40]  Steen Christensen,et al.  Bias and uncertainty in regression-calibrated models of groundwater flow in heterogeneous media , 2006 .

[41]  Rangasami L. Kashyap,et al.  Optimal Choice of AR and MA Parts in Autoregressive Moving Average Models , 1982, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[42]  Qingyun Duan,et al.  An integrated hydrologic Bayesian multimodel combination framework: Confronting input, parameter, and model structural uncertainty in hydrologic prediction , 2006 .

[43]  S. P. Neuman,et al.  Maximum likelihood Bayesian averaging of uncertain model predictions , 2003 .

[44]  Jasper A. Vrugt,et al.  Comparison of point forecast accuracy of model averaging methods in hydrologic applications , 2010 .

[45]  Ming Ye,et al.  Identification of sorption processes and parameters for radionuclide transport in fractured rock , 2012 .

[46]  Yuqiong Liu,et al.  Uncertainty in hydrologic modeling: Toward an integrated data assimilation framework , 2007 .

[47]  Hamid Moradkhani,et al.  Toward reduction of model uncertainty: Integration of Bayesian model averaging and data assimilation , 2012 .

[48]  Mary C. Hill,et al.  Sensitivity analysis, calibration, and testing of a distributed hydrological model using error‐based weighting and one objective function , 2009 .

[49]  Claire R. Tiedeman,et al.  Effect of correlated observation error on parameters, predictions, and uncertainty , 2013 .

[50]  S. P. Neuman,et al.  Sensitivity analysis and assessment of prior model probabilities in MLBMA with application to unsaturated fractured tuff , 2005 .

[51]  Jaesik Choi,et al.  IMPROVING GROUNDWATER FLOW MODEL PREDICTION USING COMPLEMENTARY DATA-DRIVEN MODELS , 2012 .

[52]  Ming Ye,et al.  MMA: A Computer Code for Multimodel Analysis , 2010 .

[53]  Mary C. Hill,et al.  MMA, A Computer Code for Multi-Model Analysis , 2014 .

[54]  David R. Anderson,et al.  Model selection and multimodel inference : a practical information-theoretic approach , 2003 .

[55]  D. Madigan,et al.  Bayesian Model Averaging in Proportional Hazard Models: Assessing the Risk of a Stroke , 1997 .

[56]  Y. Rubin,et al.  A Bayesian approach for inverse modeling, data assimilation, and conditional simulation of spatial random fields , 2010 .

[57]  J. Bredehoeft.  The conceptualization model problem—surprise , 2005 .

[58]  Richard A. Davis,et al.  Time Series: Theory and Methods , 2013 .

[59]  J. M. Bates,et al.  The Combination of Forecasts , 1969 .

[60]  Xiaobao Li,et al.  Multiple Parameterization for Hydraulic Conductivity Identification , 2008, Ground water.

[61]  H. Akaike.  A new look at the statistical model identification , 1974 .

[62]  Richard L. Cooley,et al.  Regression modeling of ground-water flow , 1990 .

[63]  Keming Yu,et al.  Bayesian Mode Regression , 2012, 1208.0579.

[64]  A. Raftery,et al.  Using Bayesian Model Averaging to Calibrate Forecast Ensembles , 2005 .

[65]  Doug Nychka,et al.  Forecasting skill of model averages , 2010 .

[66]  Jens Christian Refsgaard,et al.  Assessment of hydrological model predictive ability given multiple conceptual geological models , 2012 .

[67]  Velimir V. Vesselinov,et al.  Maximum likelihood Bayesian averaging of airflow models in unsaturated fractured tuff using Occam and variance windows , 2010 .

[68]  B. Hansen.  Least Squares Model Averaging , 2007 .

[69]  S. P. Neuman,et al.  Bayesian analysis of data-worth considering model and parameter uncertainties , 2012 .

[70]  Ming Ye,et al.  Comment on “Inverse groundwater modeling for hydraulic conductivity estimation using Bayesian model averaging and variance window” by Frank T.‐C. Tsai and Xiaobao Li , 2010 .

[71]  David W. Pollock,et al.  A Controlled Experiment in Ground Water Flow Model Calibration , 1998 .

[72]  Alain Dassargues,et al.  Conceptual model uncertainty in groundwater modeling: Combining generalized likelihood uncertainty estimation and Bayesian model averaging , 2008 .

[73]  C. Tiedeman,et al.  Effective Groundwater Model Calibration , 2007 .

[74]  Ming Ye,et al.  Dependence of Bayesian Model Selection Criteria and Fisher Information Matrix on Sample Size , 2011 .

[75]  John D Bredehoeft.  From models to performance assessment: the conceptualization problem. , 2003, Ground water.

[76]  John Doherty,et al.  Predictive error dependencies when using pilot points and singular value decomposition in groundwater model calibration , 2008 .

[77]  Frank T.-C. Tsai,et al.  Inverse groundwater modeling for hydraulic conductivity estimation using Bayesian model averaging and variance window , 2008 .

[78]  C. Tiedeman,et al.  Effective Groundwater Model Calibration: With Analysis of Data, Sensitivities, Predictions, and Uncertainty , 2007 .

[79]  G. Kuczera.  Improved parameter inference in catchment models: 1. Evaluating parameter uncertainty , 1983 .

[80]  Ming Ye,et al.  Comparing Nonlinear Regression and Markov Chain Monte Carlo Methods for Assessment of Prediction Uncertainty in Vadose Zone Modeling , 2012 .

[81]  K. Beven.  Towards a coherent philosophy for modelling the environment , 2002, Proceedings of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences.

[82]  Ming Ye,et al.  Expert elicitation of recharge model probabilities for the Death Valley regional flow system , 2008 .

[83]  Clifford M. Hurvich,et al.  Regression and time series model selection in small samples , 1989 .

[84]  Jasper A. Vrugt,et al.  Combining multiobjective optimization and Bayesian model averaging to calibrate forecast ensembles of soil hydraulic models , 2008 .

[85]  Mary C. Hill,et al.  UCODE_2005 and six other computer codes for universal sensitivity analysis, calibration, and uncertainty evaluation constructed using the JUPITER API , 2006 .