An Integrated Method for Simultaneous Calibration and Parameter Selection in Computer Models

For many large and complex computer models, there usually exist a large number of unknown parameters. To improve the computer model's predictive performance for more reliable and confident decision making, two important issues have to be addressed. The first is to calibrate the computer model and the second is to select the most influential set of parameters. However, these two issues are often addressed independently, which may waste computational effort. In this article, a Gaussian process-based Bayesian method is first proposed to simultaneously calibrate and select parameters in stochastic computer models. The whole procedure can be conducted more efficient by sharing the data information between these two steps. To further ease the computational burden, an approximation approach based on a weighted normal approximation is proposed to evaluate the posteriors in the proposed Bayesian method. A sequential approximation procedure is further proposed to improve the approximation accuracy by allocating the sequential design points more appropriately. The efficiency and accuracy of the proposed approaches are compared in a building energy model and a pandemic influenza simulation model.

[1]  Szu Hui Ng,et al.  Simulation input analysis: joint criterion for factor identification and parameter estimation , 2002, WSC '02.

[2]  R. Cheng,et al.  Sensitivity of computer simulation experiments to errors in input data , 1997 .

[3]  Agus Sudjianto,et al.  Blind Kriging: A New Method for Developing Metamodels , 2008 .

[4]  J. Berger,et al.  Objective Bayesian Analysis of Spatially Correlated Data , 2001 .

[5]  Szu Hui Ng,et al.  Joint criterion for factor identification and parameter estimation , 2002, Proceedings of the Winter Simulation Conference.

[6]  Hong Wan,et al.  A two-phase screening procedure for simulation experiments , 2009, Proceedings of the Winter Simulation Conference, 2005..

[7]  Sylvia Richardson,et al.  Markov Chain Monte Carlo in Practice , 1997 .

[8]  Wei Xie,et al.  Quantifying Input Uncertainty via Simulation Confidence Intervals , 2014, INFORMS J. Comput..

[9]  Nando de Freitas,et al.  Bayesian Optimization in a Billion Dimensions via Random Embeddings , 2013, J. Artif. Intell. Res..

[10]  Szu Hui Ng,et al.  Bayesian Kriging Analysis and Design for Stochastic Simulations , 2012, TOMC.

[11]  Szu Hui Ng,et al.  A sequential approach for stochastic computer model calibration and prediction , 2013, Reliab. Eng. Syst. Saf..

[12]  V. Roshan Joseph,et al.  Bayesian Computation Using Design of Experiments-Based Interpolation Technique , 2012, Technometrics.

[13]  H. Rue,et al.  Approximate Bayesian inference for latent Gaussian models by using integrated nested Laplace approximations , 2009 .

[14]  Sonja Kuhnt,et al.  Design and analysis of computer experiments , 2010 .

[15]  Barry L. Nelson,et al.  Single-experiment input uncertainty , 2015, J. Simulation.

[16]  Barry L. Nelson,et al.  Control variates for screening, selection, and estimation of the best , 2006, TOMC.

[17]  L. Tierney,et al.  Accurate Approximations for Posterior Moments and Marginal Densities , 1986 .

[18]  Saltelli Andrea,et al.  Global Sensitivity Analysis: The Primer , 2008 .

[19]  Stefan M. Wild,et al.  Bayesian Calibration and Uncertainty Analysis for Computationally Expensive Models Using Optimization and Radial Basis Function Approximation , 2008 .

[20]  Kenny Q. Ye,et al.  Variable Selection for Gaussian Process Models in Computer Experiments , 2006, Technometrics.

[21]  Jack P. C. Kleijnen,et al.  Application-driven sequential designs for simulation experiments: Kriging metamodelling , 2004, J. Oper. Res. Soc..

[22]  A. O'Hagan,et al.  Bayesian calibration of computer models , 2001 .

[23]  Andrew Gelman,et al.  Handbook of Markov Chain Monte Carlo , 2011 .

[24]  Paul Raftery,et al.  A review of methods to match building energy simulation models to measured data , 2014 .

[25]  Szu Hui Ng,et al.  Calibration, Validation, and Prediction in Random Simulation Models , 2015, ACM Trans. Model. Comput. Simul..

[26]  Peter Reichert,et al.  Practical identifiability of ASM2d parameters--systematic selection and tuning of parameter subsets. , 2002, Water research.

[27]  WangZiyu,et al.  Bayesian optimization in a billion dimensions via random embeddings , 2016 .

[28]  Jack P. C. Kleijnen,et al.  Kriging Metamodeling in Simulation: A Review , 2007, Eur. J. Oper. Res..

[29]  Szu Hui Ng,et al.  Calibration of Stochastic Computer Models Using Stochastic Approximation Methods , 2013, IEEE Transactions on Automation Science and Engineering.

[30]  J. S. Rao,et al.  Spike and slab variable selection: Frequentist and Bayesian strategies , 2005, math/0505633.

[31]  R. R. Hocking,et al.  Selection of the Best Subset in Regression Analysis , 1967 .

[32]  Q. J. Wang.  The Genetic Algorithm and Its Application to Calibrating Conceptual Rainfall-Runoff Models , 1991 .

[33]  D. Higdon,et al.  Computer Model Calibration Using High-Dimensional Output , 2008 .

[34]  Soroosh Sorooshian,et al.  Toward improved calibration of hydrologic models: Multiple and noncommensurable measures of information , 1998 .

[35]  Peter Z. G. Qian,et al.  Bayesian Hierarchical Modeling for Integrating Low-Accuracy and High-Accuracy Experiments , 2008, Technometrics.

[36]  Szu Hui Ng,et al.  Reducing parameter uncertainty for stochastic systems , 2006, TOMC.

[37]  Faker Zouaoui,et al.  Accounting for input-model and input-parameter uncertainties in simulation , 2004 .

[38]  Wei Chen,et al.  Bayesian Validation of Computer Models , 2009, Technometrics.

[39]  Robert Haining,et al.  Statistics for spatial data: by Noel Cressie, 1991, John Wiley & Sons, New York, 900 p., ISBN 0-471-84336-9, US $89.95 , 1993 .

[40]  Rob J Hyndman,et al.  Computing and Graphing Highest Density Regions , 1996 .

[41]  R. Cheng,et al.  Two-point methods for assessing variability in simulation output , 1998 .

[42]  Robert B. Gramacy,et al.  Ja n 20 08 Bayesian Treed Gaussian Process Models with an Application to Computer Modeling , 2009 .

[43]  E. George,et al.  Journal of the American Statistical Association is currently published by American Statistical Association. , 2007 .

[44]  Kai-Sheng Song,et al.  Goodness-of-fit tests based on Kullback-Leibler discrimination information , 2002, IEEE Trans. Inf. Theory.

[45]  Richard D. Wilkinson,et al.  Bayesian Calibration of Expensive Multivariate Computer Experiments , 2010 .

[46]  Sigrún Andradóttir,et al.  Reactive strategies for containing developing outbreaks of pandemic influenza , 2011, BMC public health.

[47]  Stephen E. Chick,et al.  Input Distribution Selection for Simulation Experiments: Accounting for Input Uncertainty , 2001, Oper. Res..

[48]  Victor Nian,et al.  A simultaneous calibration and parameter ranking method for building energy models , 2017 .

[49]  R. O’Hara,et al.  A review of Bayesian variable selection methods: what, how and which , 2009 .

[50]  George Kuczera,et al.  Probabilistic optimization for conceptual rainfall-runoff models: A comparison of the shuffled complex evolution and simulated annealing algorithms , 1999 .

[51]  A. P. Dawid,et al.  Gaussian Processes to Speed up Hybrid Monte Carlo for Expensive Bayesian Integrals , 2003 .

[52]  Wei Xie,et al.  A Bayesian Framework for Quantifying Uncertainty in Stochastic Simulation , 2014, Oper. Res..