Multiple-comparison procedures for steady-state simulations

Suppose that there are k ≥ 2 different systems (i.e., stochastic processes), where each system has an unknown steady-state mean performance and unknown asymptotic variance. We allow for the asymptotic variances to be unequal and for the distributions of the k systems to be different. We consider the problem of running independent, single-stage simulations to make multiple comparisons of the steady-state means of the different systems. We derive asymptotically valid (as the run lengths of the simulations of the systems tend to infinity) simultaneous confidence intervals for each of the following problems: all pairwise comparisons of means, all contrasts, multiple comparisons with a control and multiple comparisons with the best. Our confidence intervals are based on standardized time series methods, and we establish the asymptotic validity of each under the sole assumption that the stochastic processes representing the simulation output of the different systems satisfy a functional central limit theorem. Although simulation is the context of this paper, the results naturally apply to (asymptotically) stationary time series.

[1]  D. Slepian.  The one-sided barrier problem for Gaussian noise , 1962 .

[2]  Rupert G. Miller.  Simultaneous Statistical Inference , 1966 .

[3]  Z. Šidák.  Rectangular Confidence Regions for the Means of Multivariate Normal Distributions , 1967 .

[4]  Hayne W. Reese,et al.  Multiple comparison methods. , 1970 .

[5]  E. Spjøtvoll.  Joint confidence intervals for all linear functions of means in the one-way layout with unknown group variances , 1972 .

[6]  Paul A. Games,et al.  An Improved t Table for Simultaneous Control on g Contrasts , 1977 .

[7]  A. Tamhane.  Multiple comparisons in model i one-way anova with unequal variances , 1977 .

[8]  S. R. Dalal.  Simultaneous confidence procedures for univariate and multivariate Behrens-Fisher type problems , 1978 .

[9]  Ajit C. Tamhane,et al.  A Comparison of Procedures for Multiple Comparisons of Means with Unequal Variances , 1979 .

[10]  J. Hsu.  Simultaneous Confidence Intervals for all Distances from the "Best" , 1981 .

[11]  C. Newman,et al.  An Invariance Principle for Certain Dependent Sequences , 1981 .

[12]  Bruce W. Schmeiser,et al.  Batch Size Effects in the Analysis of Simulation Output , 1982, Oper. Res..

[13]  Lee W. Schruben,et al.  Detecting Initialization Bias in Simulation Output , 1982, Oper. Res..

[14]  Lee W. Schruben,et al.  Confidence Interval Estimation Using Standardized Time Series , 1983, Oper. Res..

[15]  J. Hsu,et al.  Multiple Comparisons with the Best Treatment , 1983 .

[16]  Paul Bratley,et al.  A guide to simulation , 1983 .

[17]  L. Schruben,et al.  Asymptotic Properties of Some Confidence Interval Estimators for Simulation Output , 1984 .

[18]  Thomas J. Santner,et al.  Design of Experiments: Ranking and Selection , 1984 .

[19]  J. Hsu.  Constrained Simultaneous Confidence Intervals for Multiple Comparisons with the Best , 1984 .

[20]  E. Carlstein.  The Use of Subseries Values for Estimating the Variance of a General Statistic from a Stationary Sequence , 1986 .

[21]  Paul Bratley,et al.  A guide to simulation (2nd ed.) , 1986 .

[22]  Linus Schrage,et al.  A guide to simulation, 2nd Edition , 1987 .

[23]  P. Glynn,et al.  Central-limit-theorem version of L = λW , 1987 .

[24]  Robert G. Sargent,et al.  Using Standardized Time Series to Estimate Confidence Intervals for the Difference Between Two Stationary Stochastic Processes , 1987, Oper. Res..

[25]  Donald L. Iglehart,et al.  Simulation Output Analysis Using Standardized Time Series , 1990, Math. Oper. Res..

[26]  Barry L. Nelson,et al.  Batch-size effects on simulation optimization using multiple comparisons with the best , 1990, 1990 Winter Simulation Conference Proceedings.

[27]  Ward Whitt,et al.  Estimating the asymptotic variance with batch means , 1991, Oper. Res. Lett..

[28]  Wei-Ning Yang,et al.  Using Common Random Numbers and Control Variates in Multiple-Comparison Procedures , 1991, Oper. Res..

[29]  A. F. Seila,et al.  Cramer-von Mises variance estimators for simulations , 1991, 1991 Winter Simulation Conference Proceedings..

[30]  Keebom Kang,et al.  Cramér-von Mises variance estimators for simulations , 1991, WSC '91.

[31]  Keebom Kang,et al.  An Investigation of Finite Sample Behavior of Confidence Interval Estimation Procedures in Computer Simulation , 1991 .

[32]  Keebom Kang,et al.  An Investigation of Finite-Sample Behavior of Confidence Interval Estimators , 1992, Oper. Res..

[33]  Barry L. Nelson.  Robust multiple comparisons under common random numbers , 1993, TOMC.

[34]  Barry L. Nelson,et al.  Control-variate models of common random numbers for multiple comparisons with the best , 1993 .

[35]  Barry L. Nelson,et al.  Multiple comparisons with the best for steady-state simulation , 1993, TOMC.

[36]  Marvin K. Nakayama,et al.  Two-stage stopping procedures based on standardized time series , 1994 .

[37]  Marvin K. Nakayama,et al.  Two-stage procedures for multiple comparisons with a control in steady-state simulations , 1996, Winter Simulation Conference.

[38]  David Goldsman,et al.  Standardized Time Series L P -Norm Variance Estimators for Simulations , 1998 .