Evaluating alternative system configurations using simulation: A nonparametric approach

The real utility of simulation lies in comparing different alternatives that might represent competing system designs. Conventional statistical techniques are not directly applicable to the analysis of simulation output data in the evaluation of competing alternatives since the usual assumptions of normality and common variance are difficult to justify in simulation experiments. This paper revisits a known nonparametric test whose application has recently become feasible due to considerable increases in computing power:randomization tests assess the significance of the observed value of the test statistic by evaluating different permutations of the data. The procedure only requires invariance of the data under all permutations.

[1]  S. L. Andersen,et al.  Permutation Theory in the Derivation of Robust Criteria and the Study of Departures from Assumption , 1955 .

[2]  Belva J. Cooley,et al.  DATA ANALYSIS FOR SIMULATION EXPERIMENTS: APPLICATION OF A DISTRIBUTION‐FREE MULTIPLE COMPARISONS PROCEDURE , 1980 .

[3]  B. L. Welch.  THE SIGNIFICANCE OF THE DIFFERENCE BETWEEN TWO MEANS WHEN THE POPULATION VARIANCES ARE UNEQUAL , 1938 .

[4]  David Goldsman,et al.  Ranking and selection in simulation , 1983, WSC '83.

[5]  M. Kendall.  Statistical Methods for Research Workers , 1937, Nature.

[6]  J. Kleijnen.  Statistical tools for simulation practitioners , 1986 .

[7]  Gordon M. Clark,et al.  Tutorial: Analysis of simulation output to compare alternatives , 1988, 1988 Winter Simulation Conference Proceedings.

[8]  David Tritchler,et al.  On Inverting Permutation Tests , 1984 .

[9]  James V. Jucker,et al.  POLICY‐COMPARING SIMULATION EXPERIMENTS: DESIGN AND ANALYSIS , 1975 .

[10]  Barry L. Nelson,et al.  Optimization over a finite number of system designs with one-stage sampling and multiple comparisons with the best , 1988, WSC '88.

[11]  H. Scheffé,et al.  The Analysis of Variance , 1960 .

[12]  Robert Tibshirani,et al.  Bootstrap Methods for Standard Errors, Confidence Intervals, and Other Measures of Statistical Accuracy , 1986 .

[13]  W. Hoeffding.  The Large-Sample Power of Tests Based on Permutations of Observations , 1952 .

[14]  Robert V. Foutz,et al.  A method for constructing exact tests from test statistics that have unknown null distributions , 1980 .

[15]  Wei-Ning Yang,et al.  A Bonferroni selection procedure when using commom random numbers with unknown variances , 1986, WSC '86.

[16]  Marcello Pagano,et al.  Efficient Calculation of the Permutation Distribution of Trimmed Means , 1991 .

[17]  Ian T. Jolliffe,et al.  Simulation Methodology for Statisticians, Operations Analysts, and Engineers, Vol. 1. , 1990 .

[18]  J. I.  The Design of Experiments , 1936, Nature.

[19]  S. Dalal,et al.  ALLOCATION OF OBSERVATIONS IN RANKING AND SELECTION WITH UNEQUAL VARIANCES , 1971 .

[20]  Averill M. Law,et al.  Simulation Modeling and Analysis , 1982 .

[21]  Edward J. Dudewicz,et al.  Statistics in simulation: How to design for selecting the best alternative , 1976, WSC '76.

[22]  M. Pagano,et al.  On Obtaining Permutation Distributions in Polynomial Time , 1983 .

[23]  B. Efron.  Bootstrap Methods: Another Look at the Jackknife , 1979 .

[24]  L. Toothaker.  Multiple Comparisons for Researchers , 1991 .

[25]  W.-N. Yang.  Optimization using common random numbers, control variates and multiple comparisons with the best , 1989, WSC '89.

[26]  Scott M. Smith,et al.  Computer Intensive Methods for Testing Hypotheses: An Introduction , 1989 .

[27]  P. Bickel,et al.  Mathematical Statistics: Basic Ideas and Selected Topics , 1977 .

[28]  R. Deal.  Simulation Modeling and Analysis (2nd Ed.) , 1994 .