Confidence Intervals for Steady-State Simulations: I. A Survey of Fixed Sample Size Procedures

We consider the problem of constructing a confidence interval for the steady-state mean of a stochastic process by means of simulation, and study the five main methods which have been proposed replication, batch means, autoregressive representation, spectrum analysis, and regeneration cycles for the case when the length of the simulation is fixed in advance. Comparing the performances of these methods on stochastic models with known steady-state means, we find that the simulator should exercise caution in interpreting the results from a simulation of fixed length, and that the length of a simulation adequate for acceptable performance is highly model-dependent. We also investigate possible sources of error for the methods, and conclude that variance estimator bias is more important than point estimator bias in confidence interval coverage degradation.

[1]  Averill M. Law,et al.  An Analytical Evaluation of Alternative Strategies in Steady-State Simulation , 1984, Oper. Res..

[2]  James R. Wilson,et al.  Evaluation of startup policies in simulation experiments , 1978 .

[3]  C. J. Ancker,et al.  Evaluation of commonly used rules for detecting “steady state” in computer simulation , 1978 .

[4]  D. Daley.  The serial correlation coefficients of waiting times in a stationary single server queue , 1968, Journal of the Australian Mathematical Society.

[5]  George S. Fishman,et al.  Solution of Large Networks by Matrix Methods , 1976, IEEE Transactions on Systems, Man, and Cybernetics.

[6]  M. S. Bartlett,et al.  Some probability limit theorems with statistical applications , 1953, Mathematical Proceedings of the Cambridge Philosophical Society.

[7]  D. Fraser.  Nonparametric methods in statistics , 1957 .

[8]  Bruce W. Schmeiser,et al.  Batch Size Effects in the Analysis of Simulation Output , 1982, Oper. Res..

[9]  George S. Fishman,et al.  Statistical Analysis for Queueing Simulations , 1973 .

[10]  Myint Tin.  Comparison of Some Ratio Estimators , 1965 .

[11]  Michael A. Crane,et al.  Simulating Stable Stochastic Systems, II: Markov Chains , 1974, JACM.

[12]  T. Anderson.  Statistical analysis of time series , 1974 .

[13]  George S. Fishman.  Estimation in Multiserver Queuing Simulations , 1974, Oper. Res..

[14]  G. S. Fishman,et al.  Spectral Methods in Econometrics. , 1970 .

[15]  D. Brillinger.  Estimation of the mean of a stationary time series by sampling , 1973, Journal of Applied Probability.

[16]  Averill M. Law,et al.  A Sequential Procedure for Determining the Length of a Steady-State Simulation , 1979, Oper. Res..

[17]  Michael A. Crane,et al.  Simulating Stable Stochastic Systems, IV: Approximation Techniques , 1975 .

[18]  Averill M. Law,et al.  Confidence Intervals for Steady-State Simulations II: A Survey of Sequential Procedures , 1982 .

[19]  Steven D. Duket,et al.  Examination of simulation output using spectral methods , 1978 .

[20]  Lee W. Schruben,et al.  Optimal Tests for Initialization Bias in Simulation Output , 1983, Oper. Res..

[21]  Philip Heidelberger,et al.  A spectral method for confidence interval generation and run length control in simulations , 1981, CACM.

[22]  G. S. Fishman.  Grouping Observations in Digital Simulation , 1978 .

[23]  James R. Wilson,et al.  A survey of research on the simulation startup problem , 1978 .

[24]  Michael A. Crane,et al.  Simulating Stable Stochastic Systems, I: General Multiserver Queues , 1974, JACM.

[25]  D. Iglehart.  Simulating stable stochastic systems, V: Comparison of ratio estimators , 1975 .

[26]  G. S. Fishman.  Principles of Discrete Event Simulation , 1978 .

[27]  Lee W. Schruben,et al.  Detecting Initialization Bias in Simulation Output , 1982, Oper. Res..

[28]  Averill M. Law,et al.  A new approach for dealing with the startup problem in discrete event simulation , 1983 .

[29]  Averill M. Law.  Confidence intervals in discrete event simulation: A comparison of replication and batch means , 1977 .

[30]  Rupert G. Miller.  The jackknife-a review , 1974 .