Validation of models: statistical techniques and data availability

This paper shows which statistical techniques can be used to validate simulation models, depending on which real-life data are available. Concerning this availability, three situations are distinguished: (i) no data; (ii) only output data; and (iii) both input and output data. In case (i)-no real data-the analysts can still experiment with the simulation model to obtain simulated data; such an experiment should be guided by the statistical theory on the design of experiments. In case (ii) only output data-real and simulated output data can be compared through the well-known two-sample Student t statistic or certain other statistics. In case (iii)-input and output data-trace-driven simulation becomes possible, but validation should not proceed in the popular way (make a scatter plot with real and simulated outputs, fit a line, and test whether that line has unit slope and passes through the origin); alternative regression and bootstrap procedures are presented. Several case studies are summarized, to illustrate the three types of situations.

[1]  James S. Hodges,et al.  Six (Or So) Things You Can Do with a Bad Model , 1991, Oper. Res..

[2]  Jack P. C. Kleijnen,et al.  Validation of Trace-Driven Simulation Models: A Novel Regression Test , 1998 .

[3]  Timothy J. Lysyk.  Stochastic Model of Eastern Spruce Budworm (Lepidoptera: Tortricidae) Phenology on White Spruce and Balsam Fir , 1989 .

[4]  Mansooreh Mollaghasemi,et al.  Validation and verification of the simulation model of a photolithography process in semiconductor manufacturing , 1998, 1998 Winter Simulation Conference. Proceedings (Cat. No.98CH36274).

[5]  Lei Rao,et al.  Development and application of a validation framework for traffic simulation models , 1998, 1998 Winter Simulation Conference. Proceedings (Cat. No.98CH36274).

[6]  Jack P. C. Kleijnen,et al.  Experimental Design for Sensitivity Analysis, Optimization and Validation of Simulation Models , 1997 .

[7]  N. J. Johnson,et al.  Modified t Tests and Confidence Intervals for Asymmetrical Populations , 1978 .

[8]  W. J. Trybula,et al.  Building simulation models without data , 1994, Proceedings of IEEE International Conference on Systems, Man and Cybernetics.

[9]  Jack P. C. Kleijnen,et al.  EUROPEAN JOURNAL OF OPERATIONAL , 1992 .

[10]  Jack P. C. Kleijnen,et al.  Searching for important factors in simulation models with many factors: Sequential bifurcation , 1997 .

[11]  Jack P. C. Kleijnen,et al.  Validation of Trace-Driven Simulation Models: Bootstrap Tests , 2001, Manag. Sci..

[12]  Jack P. C. Kleijnen,et al.  Techniques for sensitivity analysis of simulation models: A case study of the CO2 greenhouse effect , 1992, Simul..

[13]  Jack P. C. Kleijnen,et al.  A methodology for the fitting and validation of metamodels in simulation , 2000 .

[14]  J.P.C. Kleijnen,et al.  Testing the mean of an asymmetric population: Johnson's modified t test revisited , 1985 .

[15]  Jack P. C. Kleijnen,et al.  Bootstrapping and validation of metamodels in simulation , 1998, 1998 Winter Simulation Conference. Proceedings (Cat. No.98CH36274).

[16]  J. Kleijnen,et al.  Validation of trace-driven simulation models: regression analysis revisited , 1996, Proceedings Winter Simulation Conference.