Batching methods in simulation output analysis: what we know and what we don't

As an advanced tutorial, we discuss batching methods for determining point-estimator precision for steady-state simulation experiments. We emphasize batching methods in which each batch provides a point estimator analogous to that of the experiment, but we mention other methods that use batches, especially the more-general idea of standardized time series. Despite the preponderance of literature on confidence-interval estimation for the mean using adjacent nonoverlapping batches, we focus on estimating the point estimator's standard error and consider both general point estimators and general batching relationships. Literature on multivariate batching exists, but we focus on the univariate problem. We consider the initial-transient problem only in passing. Specific issues include form of the point estimator, definition of the batch statistics, form of the batch-statistics estimator, optimal batch size (including various definitions of optimal) and determining batch size. This paper is a short summary of the issues, with a fairly complete bibliography.

[1]  Averill M. Law,et al.  Confidence Intervals for Steady-State Simulations: I. A Survey of Fixed Sample Size Procedures , 1984, Oper. Res..

[2]  Halim Damerdji,et al.  Strong Consistency of the Variance Estimator in Steady-State Simulation Output Analysis , 1994, Math. Oper. Res..

[3]  Chiahon Chien,et al.  Batch size selection for the batch means method , 1994, Proceedings of Winter Simulation Conference.

[4]  M. Sherman On batch means in the simulation and statistics communities , 1995, Winter Simulation Conference Proceedings, 1995..

[5]  Bruce W. Schmeiser,et al.  Variance of the Sample Mean: Properties and Graphs of Quadratic-Form Estimators , 1993, Oper. Res..

[6]  Keebom Kang,et al.  An Investigation of Finite-Sample Behavior of Confidence Interval Estimators , 1992, Oper. Res..

[7]  Youngsoo Chun Time-Series Models of Batch-Means Processes in Simulation Analysis , 1989 .

[8]  Bruce W. Schmeiser,et al.  Properties of batch means from stationary ARMA time series , 1987 .

[9]  Barry L. Nelson,et al.  Control Variate Remedies , 1990, Oper. Res..

[10]  Averill M. Law Confidence intervals in discrete event simulation: A comparison of replication and batch means , 1977 .

[11]  Bruce W. Schmeiser,et al.  Asymptotic and Finite-Sample Correlations Between Obm Estimators , 1993, Proceedings of 1993 Winter Simulation Conference - (WSC '93).

[12]  Ward Whitt,et al.  Estimating the asymptotic variance with batch means , 1991, Oper. Res. Lett..

[13]  Bruce W. Schmeiser,et al.  SERVO: Simulation experiments with random-vector output , 1991, 1991 Winter Simulation Conference Proceedings..

[14]  Bruce W. Schmeiser,et al.  Interlaced Variance Estimators , 1993, Proceedings of 1993 Winter Simulation Conference - (WSC '93).

[15]  P. Glynn,et al.  A batch means methodology for estimation of a nonlinear function of a steady-state mean , 1997 .

[16]  Andrew F. Seila,et al.  Multivariate inference in stationary simulation using batch means , 1987, WSC '87.

[17]  John M. Chames POWER COMPARISONS FOR THE MULTIVARIATE BATCH-MEANS METHOD , 1990 .

[18]  David Goldsman,et al.  Large-Sample Results for Batch Means , 1997 .

[19]  B. Nelson,et al.  Batch size effects on the efficiency of control variates in simulation , 1989 .

[20]  B. Schmeiser,et al.  Optimal mean-squared-error batch sizes , 1995 .

[21]  Chiahon Chien Small-sample theory for steady state confidence intervals , 1988, WSC '88.

[22]  Halim Damerdji,et al.  On the batch means and area variance estimators , 1994, Proceedings of Winter Simulation Conference.

[23]  G. S. Fishman Grouping Observations in Digital Simulation , 1978 .

[24]  R. E. Schafer On assessing the precision of simulations , 1974 .

[25]  Bruce W. Schmeiser,et al.  Consistency of overlapping batch variances , 1994, Proceedings of Winter Simulation Conference.

[26]  E. Carlstein The Use of Subseries Values for Estimating the Variance of a General Statistic from a Stationary Sequence , 1986 .

[27]  Peter D. Welch,et al.  On the relationship between batch means, overlapping means and spectral estimation , 1987, WSC '87.

[28]  Thomas J. Schriber,et al.  Interactive analysis of simulation output by the method of batch means , 1979, WSC '79.

[29]  David Goldsman,et al.  Spaced batch means , 1991, Oper. Res. Lett..

[30]  朝倉 利光 R. B. Blackman and J. W. Tukey: The Measurement of Power Spectra, Dover Publications, Inc., New York, 1958, 208頁, 13.5×16cm, $1.85 , 1964 .

[31]  L. Schruben A Coverage Function for Interval Estimators of Simulation Response , 1980 .

[32]  George S. Fishman,et al.  An Implementation of the Batch Means Method , 1997, INFORMS J. Comput..

[33]  N. Adam Achieving a Confidence Interval for Parameters Estimated by Simulation , 1983 .

[34]  Chiahon Chien Small-sample theory for steady state confidence intervals , 1988, 1988 Winter Simulation Conference Proceedings.

[35]  Joseph P. Romano,et al.  BIAS‐CORRECTED NONPARAMETRIC SPECTRAL ESTIMATION , 1995 .

[36]  Averill M. Law,et al.  Confidence Intervals for Steady-State Simulations II: A Survey of Sequential Procedures , 1982 .

[37]  Lee W. Schruben,et al.  Confidence Interval Estimation Using Standardized Time Series , 1983, Oper. Res..

[38]  Michael Sherman,et al.  ON DATABASED CHOICE OF BATCH SIZE FOR SIMULATION OUTPUT ANALYSIS , 1998 .

[39]  S. Pollock,et al.  Weighted Batch Means for Confidence Intervals in Steady-State Simulations , 1993 .

[40]  D. Brillinger Estimation of the mean of a stationary time series by sampling , 1973, Journal of Applied Probability.

[41]  Bruce W. Schmeiser,et al.  Overlapping batch quantiles , 1995, WSC '95.

[42]  Averill M. Law,et al.  A Sequential Procedure for Determining the Length of a Steady-State Simulation , 1979, Oper. Res..

[43]  Jeffrey S. Simonoff,et al.  The relative importance of bias and variability in the estimation of the variance of a statistic , 1993 .

[44]  Bruce W. Schmeiser,et al.  Chapter 7 Simulation experiments , 1990 .

[45]  L. Schruben,et al.  Asymptotic Properties of Some Confidence Interval Estimators for Simulation Output , 1984 .

[46]  George S. Fishman,et al.  Problems in the statistical analysis of simulation experiments: the comparision of means and the length of sample records , 1967, CACM.

[47]  Keebom Kang,et al.  The correlation between mean and variance estimators in computer simulation , 1990 .

[48]  Bruce W. Schmeiser,et al.  Overlapping batch statistics , 1990, 1990 Winter Simulation Conference Proceedings.

[49]  Bruce W. Schmeiser,et al.  Overlapping batch means: something for nothing? , 1984, WSC '84.

[50]  森戸 晋,et al.  New directions in simulation for manufacturing and communications , 1994 .

[51]  Joseph P. Romano,et al.  On the sample variance of linear statistics derived from mixing sequences , 1993 .

[52]  Bruce W. Schmeiser,et al.  Minimal-MSE linear combinations of variance estimators of the sample mean , 1988, WSC '88.

[53]  Bruce W. Schmeiser,et al.  Graphical Methods for Evaluating and Comparing Confidence-Interval Procedures , 1990, Oper. Res..

[54]  Donald B. Percival,et al.  Three Curious Properties of the Sample Variance and Autocovariance for Stationary Processes With Unknown Mean , 1993 .

[55]  António Carvalho Pedrosa Automatic batching in simulation output analysis , 1994 .

[56]  Bruce W. Schmeiser,et al.  Batch Size Effects in the Analysis of Simulation Output , 1982, Oper. Res..

[57]  Wei-Ning Yang,et al.  Multivariate estimation and variance reduction in terminating and steady-state simulation , 1988, WSC '88.

[58]  Demet Ceylan Wood Variances and quantiles in dynamic system performance: Point estimation and standard errors , 1995 .

[59]  Bruce W. Schmeiser,et al.  Interfaced variance estimators , 1993, WSC '93.

[60]  Joseph P. Romano,et al.  The stationary bootstrap , 1994 .

[61]  M. Sherman Variance Estimation for Statistics Computed from Spatial Lattice Data , 1996 .

[62]  T. Teichmann,et al.  The Measurement of Power Spectra , 1960 .

[63]  Keebom Kang,et al.  CONFIDENCE INTERVAL ESTIMATION VIA BATCH MEANS AND TIME SERIES MODELING , 1984 .

[64]  Halim Damerdji,et al.  Mean-Square Consistency of the Variance Estimator in Steady-State Simulation Output Analysis , 1995, Oper. Res..

[65]  A. F. Seila,et al.  A Batching Approach to Quantile Estimation in Regenerative Simulations , 1982 .

[66]  L. Schruben,et al.  Properties of standardized time series weighted area variance estimators , 1990 .

[67]  Marvin K. Nakayama,et al.  Two-stage procedures for multiple comparisons with a control in steady-state simulations , 1996, Winter Simulation Conference.

[68]  Joseph P. Romano,et al.  A General Resampling Scheme for Triangular Arrays of $\alpha$-Mixing Random Variables with Application to the Problem of Spectral Density Estimation , 1992 .

[69]  Wheyming Tina Song,et al.  On the estimation of optimal batch sizes in the analysis of simulation output , 1996 .

[70]  Bruce W. Schmeiser,et al.  Correlation among estimators of the variance of the sample mean , 1987, WSC '87.

[71]  H. Damerdji,et al.  Strong consistency and other properties of the spectral variance estimator , 1991 .

[72]  Joseph P. Romano,et al.  Large Sample Confidence Regions Based on Subsamples under Minimal Assumptions , 1994 .

[73]  Richard W. Conway,et al.  Some Tactical Problems in Digital Simulation , 1963 .

[74]  Bruce W. Schmeiser,et al.  On the dispersion matrix of variance estimators of the sample mean in the analysis of simulation output , 1988 .

[75]  P. A. P. Moran,et al.  The estimation of standard errors in Monte Carlo simulation experiments , 1975 .

[76]  Barry L. Nelson,et al.  Methods for selecting the best system , 1991, WSC '91.

[77]  J. Charnes,et al.  A comparison of confidence region estimators for multivariate simulation output , 1988, 1988 Winter Simulation Conference Proceedings.

[78]  Donald L. Iglehart,et al.  Simulation Output Analysis Using Standardized Time Series , 1990, Math. Oper. Res..

[79]  Robert G. Sargent,et al.  Statistical analysis of simulation output data , 1976, SIML.