Validation of trace-driven simulation models: regression analysis revisited

For the validation of trace-driven simulation models this paper recommends a simple statistical test that uses elementary regression analysis in a novel way. This test concerns a (joint) null-hypothesis: the outputs of the simulated and the real systems have the same means and the same variances. Technically, the differences between simulated and real outputs are regressed on their sums, and the resulting slope and intercept are tested to be zero. This paper further proves that it is wrong to use a naive test that regresses the simulation outputs on the real outcomes, and hypothesizes that the resulting regression line gives a 45 /spl deg/ line through the origin. The new and the old tests are investigated in Monte Carlo experiments with inventory systems. The conclusion is that the new test has the correct type I error probability, whereas the old test (falsely) rejects a valid simulation model substantially more often than the nominal alpha level. The power of the new test increases, as the simulation model deviates more from the real system.

[1]  David G. Mayer,et al.  Regression of real-world data on model output: An appropriate overall test of validity , 1994 .

[2]  N Oreskes,et al.  Verification, Validation, and Confirmation of Numerical Models in the Earth Sciences , 1994, Science.

[3]  Jack P. C. Kleijnen,et al.  Statistical Techniques in Simulation , 1977, IEEE Transactions on Systems, Man and Cybernetics.

[4]  M. H. Hoyle,et al.  Transformations: An Introduction and a Bibliography , 1973 .

[5]  J. Banks,et al.  Discrete-Event System Simulation , 1995 .

[6]  Michael F. Kozempel,et al.  The development of the ERRC food process simulator , 1995, Simul. Pract. Theory.

[7]  Robert G. Sargent,et al.  Simulation model verification and validation , 1991, 1991 Winter Simulation Conference Proceedings..

[8]  S. K. Park,et al.  Random number generators: good ones are hard to find , 1988, CACM.

[9]  R. Cyert,et al.  Computer Models in Dynamic Economics , 1961 .

[10]  Randall P. Sadowski,et al.  Introduction to Simulation Using Siman , 1990 .

[11]  Timothy J. Lysyk.  Stochastic Model of Eastern Spruce Budworm (Lepidoptera: Tortricidae) Phenology on White Spruce and Balsam Fir , 1989 .

[12]  Averill M. Law,et al.  Simulation Modeling and Analysis , 1982 .

[13]  Jack P. C. Kleijnen,et al.  Regression metamodels and design of experiments , 1996, WSC.

[14]  S. Harrison,et al.  Regression of a model on real-system output: An invalid test of model validity , 1990 .

[15]  J. Kleijnen.  Statistical tools for simulation practitioners , 1986 .

[16]  Osman Balci,et al.  Principles of simulation model validation, verification, and testing , 1997 .

[17]  R. Larsen.  An introduction to mathematical statistics and its applications / Richard J. Larsen, Morris L. Marx , 1986 .

[18]  Y. Barlas.  Multiple tests for validation of system dynamics type of simulation models , 1989 .

[19]  P. L. Mitchell.  Misuse of regression for empirical validation of models , 1997 .

[20]  Jack P. C. Kleijnen,et al.  EUROPEAN JOURNAL OF OPERATIONAL , 1992 .

[21]  Osman Balci.  Validation, verification, and testing techniques throughout the life cycle of a simulation study , 1994, WSC '94.

[22]  Peter L. Knepell,et al.  Simulation Validation: A Confidence Assessment Methodology , 1993 .

[23]  Dennis J. Aigner,et al.  A Note on Verification of Computer Simulation Models , 1972 .

[24]  Stewart Robinson,et al.  Simulation: A Statistical Perspective , 1993 .