Validation of models: statistical techniques and data availability

This paper shows which statistical techniques can be used to validate simulation models, depending on which real-life data are available. Concerning this availability, three situations are distinguished: (i) no data; (ii) only output data; and (iii) both input and output data. In case (i)-no real data-the analysts can still experiment with the simulation model to obtain simulated data; such an experiment should be guided by the statistical theory on the design of experiments. In case (ii) only output data-real and simulated output data can be compared through the well-known two-sample Student t statistic or certain other statistics. In case (iii)-input and output data-trace-driven simulation becomes possible, but validation should not proceed in the popular way (make a scatter plot with real and simulated outputs, fit a line, and test whether that line has unit slope and passes through the origin); alternative regression and bootstrap procedures are presented. Several case studies are summarized, to illustrate the three types of situations.

[1]  M. B. Beck,et al.  On the problem of model validation for predictive exposure assessments , 1997 .

[2]  W. J. Trybula,et al.  Building simulation models without data , 1994, Proceedings of IEEE International Conference on Systems, Man and Cybernetics.

[3]  Jack P. C. Kleijnen,et al.  Techniques for sensitivity analysis of simulation models: A case study of the CO2 greenhouse effect , 1992, Simul..

[4]  Saul I. Gass,et al.  An Assessment Procedure for Simulation Models: A Case Study , 1991, Oper. Res..

[5]  Marco A. Janssen,et al.  Global Modelling: Managing Uncertainty Complexity and Incomplete Information , 1999 .

[6]  Jack P. C. Kleijnen,et al.  Validation of Trace-Driven Simulation Models: Bootstrap Tests , 2001, Manag. Sci..

[7]  Jack P. C. Kleijnen,et al.  Searching for important factors in simulation models with many factors: Sequential bifurcation , 1997 .

[8]  B. M. Brown,et al.  Practical Non-Parametric Statistics. , 1981 .

[9]  Peter L. Knepell,et al.  Simulation Validation: A Confidence Assessment Methodology , 1993 .

[10]  Jack P. C. Kleijnen,et al.  A methodology for the fitting and validation of metamodels in simulation , 2000 .

[11]  Lei Rao,et al.  Development and application of a validation framework for traffic simulation models , 1998, 1998 Winter Simulation Conference. Proceedings (Cat. No.98CH36274).

[12]  J. Kleijnen Statistical tools for simulation practitioners , 1986 .

[13]  Timothy J. Lysyk Stochastic Model of Eastern Spruce Budworm (Lepidoptera: Tortricidae) Phenology on White Spruce and Balsam Fir , 1989 .

[14]  Averill M. Law,et al.  Simulation Modeling and Analysis , 1982 .

[15]  R. Tibshirani,et al.  An introduction to the bootstrap , 1993 .

[16]  Mansooreh Mollaghasemi,et al.  Validation and verification of the simulation model of a photolithography process in semiconductor manufacturing , 1998, 1998 Winter Simulation Conference. Proceedings (Cat. No.98CH36274).

[17]  Michael F. Kozempel,et al.  The development of the ERRC food process simulator , 1995, Simul. Pract. Theory.

[18]  James S. Hodges,et al.  Six (Or So) Things You Can Do with a Bad Model , 1991, Oper. Res..

[19]  Robert J. Antonio Searching for Justice , 1988, Telos.

[20]  J.P.C. Kleijnen,et al.  Statistical validation of simulation, including case studies , 1999 .

[21]  J. van der Zouwen,et al.  Towards a Methodology for the Empirical Testing of Complex Social Cybernetics Models , 2001 .

[22]  J. C. Helton,et al.  Statistical Analyses of Scatterplots to Identify Important Factors in Large-Scale Simulations, 1: Review and Comparison of Techniques , 1999 .

[23]  Jack P. C. Kleijnen,et al.  Experimental Design for Sensitivity Analysis, Optimization and Validation of Simulation Models , 1997 .

[24]  Linda Weiser Friedman,et al.  The Simulation Metamodel , 1995 .

[25]  Jack P. C. Kleijnen,et al.  Maximizing the Simulation Output: A Competition , 1999, Simul..

[26]  Jon C. Helton,et al.  Performance Assessment for the Waste Isolation Pilot Plant: From Regulation to Calculation for 40 CFR 191.13 , 1997, Oper. Res..

[27]  Jack P. C. Kleijnen,et al.  Bootstrapping and validation of metamodels in simulation , 1998, 1998 Winter Simulation Conference. Proceedings (Cat. No.98CH36274).

[28]  J. Kleijnen,et al.  Validation of models: statistical techniques and data availability , 1999, WSC'99. 1999 Winter Simulation Conference Proceedings. 'Simulation - A Bridge to the Future' (Cat. No.99CH37038).

[29]  Jack P. C. Kleijnen,et al.  Validation of Trace-Driven Simulation Models: A Novel Regression Test , 1998 .

[30]  N. J. Johnson,et al.  Modified t Tests and Confidence Intervals for Asymmetrical Populations , 1978 .

[31]  J.P.C. Kleijnen,et al.  Testing the mean of an asymmetric population: Johnson's modified t test revisited , 1985 .

[32]  Jack P. C. Kleijnen,et al.  Validation of trace-driven simulation models: regression analysis revisited , 1996, Winter Simulation Conference.

[33]  Bernard P. Zeigler,et al.  Theory of Modelling and Simulation , 1979, IEEE Transactions on Systems, Man, and Cybernetics.

[34]  Jack P. C. Kleijnen,et al.  Statistical validation of simulation models , 1995 .

[35]  Jack P. C. Kleijnen,et al.  EUROPEAN JOURNAL OF OPERATIONAL , 1992 .

[36]  Bernard P. Zeigler,et al.  Theory of Modelling and Simulation , 1979, IEEE Transactions on Systems, Man and Cybernetics.

[37]  Jack P. C. Kleijnen,et al.  A methodology for fitting and validating metamodels in simulation , 2000, Eur. J. Oper. Res..