Practical Use of Computationally Frugal Model Analysis Methods

Three challenges compromise the utility of mathematical models of groundwater and other environmental systems: (1) a dizzying array of model analysis methods and metrics make it difficult to compare evaluations of model adequacy, sensitivity, and uncertainty; (2) the high computational demands of many popular model analysis methods (requiring 1000's, 10,000 s, or more model runs) make them difficult to apply to complex models; and (3) many models are plagued by unrealistic nonlinearities arising from the numerical model formulation and implementation. This study proposes a strategy to address these challenges through a careful combination of model analysis and implementation methods. In this strategy, computationally frugal model analysis methods (often requiring a few dozen parallelizable model runs) play a major role, and computationally demanding methods are used for problems where (relatively) inexpensive diagnostics suggest the frugal methods are unreliable. We also argue in favor of detecting and, where possible, eliminating unrealistic model nonlinearities-this increases the realism of the model itself and facilitates the application of frugal methods. Literature examples are used to demonstrate the use of frugal methods and associated diagnostics. We suggest that the strategy proposed in this paper would allow the environmental sciences community to achieve greater transparency and falsifiability of environmental models, and obtain greater scientific insight from ongoing and future modeling efforts.

[1]  M. Clark,et al.  Ancient numerical daemons of conceptual hydrological modeling: 2. Impact of time stepping schemes on model analysis and prediction , 2010 .

[2]  C. Tiedeman,et al.  Effective Groundwater Model Calibration , 2007 .

[3]  R. Spear Eutrophication in peel inlet—II. Identification of critical uncertainties via generalized sensitivity analysis , 1980 .

[4]  D. M. Ely,et al.  A method for evaluating the importance of system state observations to model predictions, with application to the Death Valley regional groundwater flow system , 2004 .

[5]  Saltelli Andrea,et al.  Global Sensitivity Analysis: The Primer , 2008 .

[6]  Joseph R. Kasprzyk,et al.  Evolutionary multiobjective optimization in water resources: The past, present, and future , 2012 .

[7]  A. Saltelli,et al.  A quantitative model-independent method for global sensitivity analysis of model output , 1999 .

[8]  K. Beven,et al.  Decision tree for choosing an uncertainty analysis methodology: a wiki experiment http://www.floodrisknet.org.uk/methods http://www.floodrisk.net , 2006 .

[9]  George Kuczera,et al.  Model smoothing strategies to remove microscale discontinuities and spurious secondary optima in objective functions in hydrological calibration , 2007 .

[10]  C. Tiedeman,et al.  Methods for using groundwater model predictions to guide hydrogeologic data collection, with application to the Death Valley regional groundwater flow system , 2003 .

[11]  John Doherty,et al.  A short exploration of structural noise , 2010 .

[12]  Claire R. Tiedeman,et al.  OPR-PPR, a Computer Program for Assessing Data Importance to Model Predictions Using Linear Statistics , 2014 .

[13]  Claire R. Tiedeman,et al.  UCODE_2014, with new capabilities to define parameters unique to predictions, calculate w eights using simulated values, estimate parameters w ith SVD, evaluate uncertainty with MCMC, and more , 2014 .

[14]  Mary C. Hill,et al.  Evaluating model structure adequacy: The case of the Maggia Valley groundwater system, southern Switzerland , 2013 .

[15]  John Doherty,et al.  Two statistics for evaluating parameter identifiability and error reduction , 2009 .

[16]  Daniel M. Tartakovsky,et al.  Assessment and management of risk in subsurface hydrology: A review and perspective , 2013 .

[17]  Mary C. Hill,et al.  Distributed Evaluation of Local Sensitivity Analysis (DELSA), with application to hydrologic models , 2014 .

[18]  E. Borgonovo Measuring Uncertainty Importance: Investigation and Comparison of Alternative Approaches , 2006, Risk analysis : an official publication of the Society for Risk Analysis.

[19]  Mary C. Hill,et al.  Comment on ``Two statistics for evaluating parameter identifiability and error reduction'' by John Doherty and Randall J. Hunt , 2010 .

[20]  Mary C Hill,et al.  Parameter and observation importance in modelling virus transport in saturated porous media-investigations in a homogenous system. , 2005, Journal of contaminant hydrology.

[21]  Patrick M. Reed,et al.  Time‐varying sensitivity analysis clarifies the effects of watershed model formulation on model behavior , 2013 .

[22]  N Oreskes,et al.  Verification, Validation, and Confirmation of Numerical Models in the Earth Sciences , 1994, Science.

[23]  A. O'Hagan,et al.  Probabilistic sensitivity analysis of complex models: a Bayesian approach , 2004 .

[24]  S. Weisberg,et al.  Residuals and Influence in Regression , 1982 .

[25]  R. L. Cooley A theory for modeling ground-water flow in heterogeneous media , 2004 .

[26]  C. Tiedeman,et al.  Effective Groundwater Model Calibration: With Analysis of Data, Sensitivities, Predictions, and Uncertainty , 2007 .

[27]  Mary C. Hill,et al.  MMA, A Computer Code for Multi-Model Analysis , 2014 .

[28]  D. G. Watts,et al.  Relative Curvature Measures of Nonlinearity , 1980 .

[29]  Wei Gong,et al.  Assessing parameter importance of the Common Land Model based on qualitative and quantitative sensitivity analysis , 2013 .

[30]  M. Kendall,et al.  The Logic of Scientific Discovery. , 1959 .

[31]  Bryan A. Tolson,et al.  Reducing the computational cost of automatic calibration through model preemption , 2010 .

[32]  Christian D Langevin,et al.  Quantifying Data Worth Toward Reducing Predictive Uncertainty , 2010, Ground water.

[33]  Mario L. V. Martina,et al.  Flood forecasting using a fully distributed model: application of the TOPKAPI model to the Upper Xixian Catchment , 2005 .

[34]  S. P. Neuman,et al.  Estimation of Aquifer Parameters Under Transient and Steady State Conditions: 1. Maximum Likelihood Method Incorporating Prior Information , 1986 .

[35]  K. Shuler,et al.  Nonlinear sensitivity analysis of multiparameter model systems , 1977 .

[36]  F. Pappenberger,et al.  Ignorance is bliss: Or seven reasons not to use uncertainty analysis , 2006 .

[37]  R. Hunt,et al.  Are Models Too Simple? Arguments for Increased Parameterization , 2007, Ground water.

[38]  J. C. Ramírez,et al.  Estimation of aquifer parameters under transient and steady-state conditions , 1984 .

[39]  N. Draper,et al.  Applied Regression Analysis , 1966 .

[40]  N. Draper,et al.  Applied Regression Analysis: Draper/Applied Regression Analysis , 1998 .

[41]  C. Tiedeman,et al.  Knowledge, transparency, and refutability in groundwater models, an example from the Death Valley regional groundwater flow system , 2013 .

[42]  John Doherty,et al.  Response to Comment on “Two statistics for evaluating parameter identifiability and error reduction” , 2010 .

[43]  Ming Ye,et al.  Towards a comprehensive assessment of model structural adequacy , 2012 .

[44]  M. Ek,et al.  Hyperresolution global land surface modeling: Meeting a grand challenge for monitoring Earth's terrestrial water , 2011 .

[45]  Joern Birkmann,et al.  World Risk Report 2011 , 2014 .

[46]  W. Hays Applied Regression Analysis. 2nd ed. , 1981 .

[47]  Dmitri Kavetski,et al.  Pursuing the method of multiple working hypotheses for hydrological modeling , 2011 .

[48]  Mary C Hill,et al.  Simulation of Water‐Table Aquifers Using Specified Saturated Thickness , 2015, Ground water.

[49]  L. Dicks,et al.  Bees, lies and evidence-based policy , 2013, Nature.

[50]  Lothar Thiele,et al.  Multiobjective Optimization Using Evolutionary Algorithms - A Comparative Case Study , 1998, PPSN.

[51]  Keith J. Beven,et al.  Environmental Modelling: An Uncertain Future?: An Introduction to Techniques for Uncertainty Estimation in Environmental Prediction , 2010 .

[52]  Phillip I. Good,et al.  Resampling Methods: A Practical Guide to Data Analysis , 2005 .

[53]  Ning Liu,et al.  Inverse Theory for Petroleum Reservoir Characterization and History Matching , 2008 .

[54]  P. Reed,et al.  Hydrology and Earth System Sciences Discussions Comparing Sensitivity Analysis Methods to Advance Lumped Watershed Model Identification and Evaluation , 2022 .

[55]  P. Reed,et al.  Beyond optimality: Multistakeholder robustness tradeoffs for regional water portfolio planning under deep uncertainty , 2014 .

[56]  Clifford H. Thurber,et al.  Parameter estimation and inverse problems , 2005 .

[57]  Gary James Jason,et al.  The Logic of Scientific Discovery , 1988 .

[58]  Nicholas Stern,et al.  Uncertainty in science and its role in climate policy , 2011, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[59]  A. Samarov Exploring Regression Structure Using Nonparametric Functional Estimation , 1993 .

[60]  S. Sorooshian,et al.  Effective and efficient global optimization for conceptual rainfall‐runoff models , 1992 .

[61]  Jack P. C. Kleijnen,et al.  Sensitivity analysis of simulation models: an overview , 2010 .

[62]  Bryan A. Tolson,et al.  Review of surrogate modeling in water resources , 2012 .

[63]  Mary C. Hill,et al.  Two-dimensional advective transport in ground-water flow parameter estimation , 1996 .

[64]  Patrick M. Reed,et al.  Technical Note: Method of Morris effectively reduces the computational demands of global sensitivity analysis for distributed watershed models , 2013 .

[65]  Ming Ye,et al.  Analysis of regression confidence intervals and Bayesian credible intervals for uncertainty quantification , 2012 .

[66]  J. Doherty,et al.  A hybrid regularized inversion methodology for highly parameterized environmental models , 2005 .

[67]  D. Helsel,et al.  Statistical methods in water resources , 2020, Techniques and Methods.

[68]  R Fisher,et al.  Design of Experiments , 1936 .

[69]  Mary C. Hill,et al.  Sensitivity analysis, calibration, and testing of a distributed hydrological model using error‐based weighting and one objective function , 2009 .

[70]  L Foglia,et al.  Testing Alternative Ground Water Models Using Cross‐Validation and Other Methods , 2007, Ground water.