To batch or not to batch?

When designing steady-state computer simulation experiments, one may be faced with the choice of batching observations in one long run or replicating a number of smaller runs. Both methods are potentially useful in the course of undertaking simulation output analysis. The tradeoffs between the two alternatives are well known: batching ameliorates the effects of initialization bias, but produces batch means that might be correlated; replication yields independent sample means, but may suffer from initialization bias at the beginning of each of the runs. We present several new results and specific examples to lend insight as to when one method might be preferred over the other. In steady-state, batching and replication perform similarly in terms of estimating the mean and variance parameter, but replication tends to do better than batching with regard to the performance of confidence intervals for the mean. Such a victory for replication may be hollow, for in the presence of an initial transient, batching often performs better than replication when it comes to point and confidence-interval estimation of the steady-state mean. We conclude---like other classic references---that in the context of estimation of the steady-state mean, batching is typically the wiser approach.

[1]  Philip Heidelberger,et al.  Analysis of initial transient deletion for replicated steady-state simulations , 1991, Oper. Res. Lett..

[2]  J. Patel,et al.  Handbook of the normal distribution , 1983 .

[3]  James R. Wilson,et al.  Evaluation of startup policies in simulation experiments , 1978 .

[4]  Averill M. Law,et al.  Simulation Modeling and Analysis , 1982 .

[5]  Keebom Kang,et al.  An Investigation of Finite-Sample Behavior of Confidence Interval Estimators , 1992, Oper. Res..

[6]  B. Schmeiser,et al.  Optimal mean-squared-error batch sizes , 1995 .

[7]  Averill M. Law,et al.  A new approach for dealing with the startup problem in discrete event simulation , 1983 .

[8]  W. David Kelton,et al.  Random Initialization Methods in Simulation , 1989 .

[9]  Ward Whitt,et al.  Estimating the asymptotic variance with batch means , 1991, Oper. Res. Lett..

[10]  Chiahon Chien Small-sample theory for steady state confidence intervals , 1988, WSC '88.

[11]  Sigrún Andradóttir,et al.  Replicated batch means for steady-state simulations , 2006 .

[12]  C. J. Ancker,et al.  Evaluation of commonly used rules for detecting “steady state” in computer simulation , 1978 .

[13]  J. J. Swain,et al.  Tests for transient means in simulated time series , 1994 .

[14]  Averill M. Law,et al.  An Analytical Evaluation of Alternative Strategies in Steady-State Simulation , 1984, Oper. Res..

[15]  Natalie M. Steiger,et al.  Output analysis: ASAP2: an improved batch means procedure for simulation output analysis , 2002, WSC '02.

[16]  R. E. Wheeler Statistical distributions , 1983, APLQ.

[17]  W. Whitt The efficiency of one long run versus independent replication in steady-state simulation , 1991 .

[18]  M. Evans Statistical Distributions , 2000 .

[19]  George S. Fishman,et al.  LABATCH.2: software for statistical analysis of simulation sample path data , 1998, 1998 Winter Simulation Conference. Proceedings (Cat. No.98CH36274).

[20]  Donald L. Iglehart,et al.  Simulation Output Analysis Using Standardized Time Series , 1990, Math. Oper. Res..

[21]  B D Titus Modified Confidence Intervals for the Mean of an Autoregressive Process. , 1985 .

[22]  George S. Fishman,et al.  Discrete-Event Simulation : Modeling, Programming, and Analysis , 2001 .

[23]  George Vassilacopoulos Testing for initialization bias in simulation output , 1989, Simul..

[24]  C. Alexopoulos,et al.  To batch or not to batch , 2003, Proceedings of the 2003 Winter Simulation Conference, 2003..

[25]  J. Neumann Distribution of the Ratio of the Mean Square Successive Difference to the Variance , 1941 .

[26]  S. R. Searle Linear Models , 1971 .

[27]  D. Sengupta Linear models , 2003 .

[28]  Sigrún Andradóttir,et al.  Replicated batch means variance estimators in the presence of an initial transient , 2006, TOMC.

[29]  Keebom Kang,et al.  The correlation between mean and variance estimators in computer simulation , 1990 .

[30]  Bruce W. Schmeiser,et al.  Overlapping batch means: something for nothing? , 1984, WSC '84.

[31]  James R. Wilson,et al.  A survey of research on the simulation startup problem , 1978 .

[32]  Lee W. Schruben,et al.  Optimal Tests for Initialization Bias in Simulation Output , 1983, Oper. Res..

[33]  Kenneth W. Bauer,et al.  Initial data truncation for univariate output of discrete-event simulations using the Kalman filter , 1996 .

[34]  Bruce W. Schmeiser,et al.  On the MSE robustness of batching estimators , 2001, Proceeding of the 2001 Winter Simulation Conference (Cat. No.01CH37304).

[35]  James R. Wilson,et al.  Convergence Properties of the Batch Means Method for Simulation Output Analysis , 2001, INFORMS J. Comput..

[36]  George S. Fishman,et al.  An Implementation of the Batch Means Method , 1997, INFORMS J. Comput..

[37]  Bruce W. Schmeiser,et al.  On choosing a single criterion for confidence-interval procedures , 2002, Proceedings of the Winter Simulation Conference.

[38]  G. Reinsel,et al.  Introduction to Mathematical Statistics (4th ed.). , 1980 .

[39]  George S. Fishman,et al.  Discrete-event simulation , 2001 .

[40]  Robert V. Hogg,et al.  Introduction to Mathematical Statistics. , 1966 .

[41]  David Goldsman,et al.  Large-Sample Results for Batch Means , 1997 .

[42]  E. Carlstein The Use of Subseries Values for Estimating the Variance of a General Statistic from a Stationary Sequence , 1986 .

[43]  C. Read,et al.  Handbook of the Normal Distribution, 2nd Edition. , 1998 .