Validation of Trace-Driven Simulation Models: Bootstrap Tests

Trace-driven or correlated inspection simulation means that the simulated and the real systems have some common inputs say, historical arrival times so that the two systems' outputs are cross-correlated. To validate such a simulation, this paper focuses on the difference between the average simulated and real responses. To evaluate this validation statistic, the paper develops a novel bootstrap technique-based on replicated runs. This validation statistic and the bootstrap technique are evaluated in extensive Monte Carlo experiments with specific single-server queues. These experiments show acceptable Type-I and Type-II error probabilities.

[1]  K. Singh,et al.  On the Asymptotic Accuracy of Efron's Bootstrap , 1981 .

[2]  Jack P. C. Kleijnen,et al.  Statistical Analysis of Random Simulations: Bootstrap Tutorial , 2002 .

[3]  Bruce W. Schmeiser,et al.  Batch Size Effects in the Analysis of Simulation Output , 1982, Oper. Res..

[4]  Jack P. C. Kleijnen,et al.  Validation of Trace-Driven Simulation Models: A Novel Regression Test , 1998 .

[5]  E. Ziegel,et al.  Bootstrapping: A Nonparametric Approach to Statistical Inference , 1993 .

[6]  B. K. Ghosh,et al.  Handbook of sequential analysis , 1991 .

[7]  J. Shao,et al.  The jackknife and bootstrap , 1996 .

[8]  L. Breiman.  The Little Bootstrap and other Methods for Dimensionality Selection in Regression: X-Fixed Prediction Error , 1992 .

[9]  Jack P. C. Kleijnen,et al.  A methodology for fitting and validating metamodels in simulation , 2000, Eur. J. Oper. Res..

[10]  Averill M. Law,et al.  Simulation Modeling and Analysis , 1982 .

[11]  S. T. Buckland,et al.  An Introduction to the Bootstrap. , 1994 .

[12]  P. Hall.  The Bootstrap and Edgeworth Expansion , 1992 .

[13]  Jack P. C. Kleijnen,et al.  Validation of trace-driven simulation models: more on bootstrap tests , 2000, 2000 Winter Simulation Conference Proceedings (Cat. No.00CH37165).

[14]  Pierre L'Ecuyer,et al.  Good Parameters and Implementations for Combined Multiple Recursive Random Number Generators , 1999, Oper. Res..

[15]  Jack P. C. Kleijnen,et al.  Strategic directions in verification, validation, and accreditation research , 2000, 2000 Winter Simulation Conference Proceedings (Cat. No.00CH37165).

[16]  D. Freedman,et al.  Some Asymptotic Theory for the Bootstrap , 1981 .

[17]  J. Haddock,et al.  The threshold bootstrap: a new approach to simulation output analysis , 1993, WSC '93.

[18]  V. Statulevičius,et al.  Limit Theorems of Probability Theory , 2000 .

[19]  R. Cheng,et al.  Sensitivity of computer simulation experiments to errors in input data , 1997 .

[20]  Michael Sherman.  On batch means in the simulation and statistics communities , 1995, WSC '95.

[21]  L.W.G. Strijbosch,et al.  New Proposals for the Validation of Trace-Driven Simulations , 1997 .

[22]  Jack P. C. Kleijnen,et al.  Validation of models: statistical techniques and data availability , 1999, WSC '99.