Quantifying Input Uncertainty via Simulation Confidence Intervals

We consider the problem of deriving confidence intervals for the mean response of a system that is represented by a stochastic simulation whose parametric input models have been estimated from “real-world” data. As opposed to standard simulation confidence intervals, we provide confidence intervals that account for uncertainty about the input model parameters; our method is appropriate when enough simulation effort can be expended to make simulation-estimation error relatively small. To achieve this we introduce metamodel-assisted bootstrapping that propagates input variability through to the simulation response via an equation-based model rather than by simulating. We develop a metamodel strategy and associated experiment design method that avoid the need for low-order approximation to the response and that minimizes the impact of intrinsic (simulation) error on confidence level accuracy. Asymptotic analysis and empirical tests over a wide range of simulation effort show that confidence intervals obtaine...

[1]  Szu Hui Ng,et al.  Reducing input parameter uncertainty for simulations , 2001, Proceeding of the 2001 Winter Simulation Conference (Cat. No.01CH37304).

[2]  J. Shao,et al.  The jackknife and bootstrap , 1996 .

[3]  Szu Hui Ng,et al.  Simulation input analysis: joint criterion for factor identification and parameter estimation , 2002, WSC '02.

[4]  Jack P. C. Kleijnen,et al.  Sensitivity analysis versus uncertainty analysis: when to use what? in predictability and nonlinear , 1994 .

[5]  Stephen E. Chick.  Bayesian methods: bayesian methods for simulation , 2000, WSC '00.

[6]  J. Kleijnen.  Statistical tools for simulation practitioners , 1986 .

[7]  R. Cheng,et al.  Sensitivity of computer simulation experiments to errors in input data , 1997 .

[8]  Peter W. Glynn,et al.  Computing the distribution function of a conditional expectation via Monte Carlo: discrete conditioning spaces , 1999, WSC '99.

[9]  Barry L. Nelson,et al.  Stochastic kriging for simulation metamodeling , 2008, WSC 2008.

[10]  Peter W. Glynn,et al.  Computing the distribution function of a conditional expectation via Monte Carlo: discrete conditioning spaces , 1999, WSC '99.

[11]  James R. Wilson,et al.  Accounting for Parameter Uncertainty in Simulation Input Modeling , 2003 .

[12]  Lee W. Schruben,et al.  Resampling methods for input modeling , 2001, Proceeding of the 2001 Winter Simulation Conference (Cat. No.01CH37304).

[13]  Szu Hui Ng,et al.  Joint criterion for factor identification and parameter estimation , 2002, Proceedings of the Winter Simulation Conference.

[14]  Shane G. Henderson.  Input model uncertainty: why do we care and what should we do about it? , 2003, Proceedings of the 2003 Winter Simulation Conference, 2003..

[15]  Lee W. Schruben,et al.  Uniform and bootstrap resampling of empirical distributions , 1993, WSC '93.

[16]  Thomas J. Santner,et al.  The Design and Analysis of Computer Experiments , 2003, Springer Series in Statistics.

[17]  Shane G. Henderson,et al.  A kernel approach to estimating the density of a conditional expectation , 2003, Proceedings of the 2003 Winter Simulation Conference, 2003..

[18]  Stephen E. Chick.  Steps to implement Bayesian input distribution selection , 1999, WSC '99.

[19]  M. Farooq,et al.  Note on the generation of random points uniformly distributed in hyper-ellipsoids , 2002, Proceedings of the Fifth International Conference on Information Fusion. FUSION 2002. (IEEE Cat.No.02EX5997).

[20]  Canan G. Corlu,et al.  Accounting for Parameter Uncertainty in Large-Scale Stochastic Simulations with Correlated Inputs , 2011, Oper. Res..

[21]  Averill M. Law,et al.  Simulation modelling and analysis , 1991 .

[22]  Russell C. H. Cheng,et al.  Calculation of confidence intervals for simulation output , 2004, TOMC.

[23]  Stephen E. Chick,et al.  Input Distribution Selection for Simulation Experiments: Accounting for Input Uncertainty , 2001, Oper. Res..

[24]  Stephen E. Chick,et al.  Bayesian methods for simulation , 2000, 2000 Winter Simulation Conference Proceedings (Cat. No.00CH37165).

[25]  Wei Xie,et al.  A framework for input uncertainty analysis , 2010, Proceedings of the 2010 Winter Simulation Conference.

[26]  James R. Wilson,et al.  Accounting for input model and parameter uncertainty in simulation , 2001, Proceeding of the 2001 Winter Simulation Conference (Cat. No.01CH37304).

[27]  Russell C. H. Cheng,et al.  Selecting input models , 1994, Proceedings of Winter Simulation Conference.

[28]  Lawrence Leemis,et al.  Panel on current issues in simulation input modeling , 2002, Proceedings of the Winter Simulation Conference.

[29]  Shane G. Henderson,et al.  Input modeling: input model uncertainty: why do we care and what should we do about it? , 2003, WSC '03.

[30]  Michael Freimer,et al.  Collecting data and estimating parameters for input distributions , 2002, Proceedings of the Winter Simulation Conference.

[31]  S. Chick.  Bayesian Analysis For Simulation Input And Output , 1997, Winter Simulation Conference Proceedings,.

[32]  Faker Zouaoui,et al.  Accounting for input-model and input-parameter uncertainties in simulation , 2004 .

[33]  S. Henderson,et al.  A kernel approach to estimating the density of a conditional expectation , 2003 .

[34]  R. Cheng,et al.  Two-point methods for assessing variability in simulation output , 1998 .

[35]  Xi Chen,et al.  The effects of common random numbers on stochastic kriging metamodels , 2012, TOMC.

[36]  Russell R. Barton.  A MORE COMPLETE CHARACTERIZATION OF UNCERTAINTY : CAN IT BE DONE ? , 2007 .