Heuristics for the regression of stochastic simulations

Modelling and simulation environments that are stochastic in nature present a multitude of problems in the creation of meta-models, notably the reduction of the quality of fit due to statistical error. This report addresses this issue by first determining the minimum number of repetitions required for interval estimates to hold true regardless of interval width. These intervals are then used for meta-model regressions. Four measures were examined. Sample mean and variance interval coverage accuracy was studied by varying skewness, kurtosis, desired coverage, and sample size within the Pearson family of distributions. Binomial proportion and quantile interval coverage accuracy was studied with the standard normal with varying sample size, desired coverage, and quantile level. Finally, heuristic measures to determine how repetitions are related to the quality of the meta-model fit were developed based on the experimentation with a canonical problem. The ratio of confidence interval width to the range of sample measures was found to be an indicator of the impact of statistical error on the quality of model fit. Regression methods of weighted least squares (WLS), ordinary least squares for constant sample sizes, and constant interval widths were compared. The WLS method is suggested for stochastic regressions of simulations.

[1]  E. B. Wilson.  Probable Inference, the Law of Succession, and Statistical Inference , 1927 .

[2]  E. Ziegel,et al.  Bootstrapping: A Nonparametric Approach to Statistical Inference , 1993 .

[3]  M. Kenward,et al.  An Introduction to the Bootstrap , 2007 .

[4]  Robert Gould,et al.  A Modern Approach to Regression with R , 2010 .

[5]  André I. Khuri,et al.  Response surface methodology , 2010 .

[6]  E. Jack Chen,et al.  Metamodels for Estimating Quantiles of Systems with One Controllable Parameter , 2009, Simul..

[7]  L. F. Perrone,et al.  APPLICATIONS OF DISCRETE-EVENT SIMULATION TO SUPPORT MANUFACTURING LOGISTICS DECISION-MAKING : A SURVEY , 2006 .

[8]  A. Agresti,et al.  Approximate is Better than “Exact” for Interval Estimation of Binomial Proportions , 1998 .

[9]  George E. P. Box,et al.  Empirical Model‐Building and Response Surfaces , 1988 .

[10]  Daniel P. Schrage,et al.  GENERATING DYNAMIC MODELS INCLUDING UNCERTAINTY FOR USE IN AIRCRAFT CONCEPTUAL DESIGN , 1997 .

[11]  A. Law,et al.  Relative width sequential confidence intervals for the mean , 1981 .

[12]  Dimitri N. Mavris,et al.  Aerodynamic Surrogate Modeling of Variable Geometry , 2012 .

[13]  Michael J. North,et al.  Tutorial on agent-based modelling and simulation , 2005, Proceedings of the Winter Simulation Conference, 2005..

[14]  L. Brown,et al.  Interval Estimation for a Binomial Proportion , 2001 .

[15]  Jan Ola Strandhagen,et al.  Applications of Discrete-Event Simulation to Support Manufacturing Logistics Decision-Making: A Survey , 2006, Proceedings of the 2006 Winter Simulation Conference.