Verification and validation of simulation models

In this paper we discuss verification and validation of simulation models. Four different approaches to deciding model validity are described; two different paradigms that relate verification and validation to the model development process are presented; various validation techniques are defined; conceptual model validity, model verification, operational validity, and data validity are discussed; a way to document results is given; a recommended procedure for model validation is presented; and model accreditation is briefly discussed.

[1]  I ÖrenTuncer Concepts and criteria to assess acceptability of simulation studies , 1981 .

[2]  Osman Balci,et al.  A methodology for certification of modeling and simulation applications , 2001, TOMC.

[3]  Dale K. Pace Naval modeling and simulation verification, validation, and accreditation , 1993, WSC '93.

[4]  Lee W. Schruben,et al.  Establishing the credibility of simulations , 1980 .

[5]  Saul I. Gass,et al.  Letter to the Editor - Guidelines for Model Evaluation: An Abridged Version of the U.S. General Accounting Office Exposure Draft , 1980, Oper. Res..

[6]  Bernard P. Zeigler,et al.  Theory of Modelling and Simulation , 1979, IEEE Transactions on Systems, Man, and Cybernetics.

[7]  Osman Balci,et al.  How To Assess The Acceptability And Credibility Of Simulation Results , 1989, 1989 Winter Simulation Conference Proceedings.

[8]  George S. Fishman,et al.  On validation of simulation models , 1973, AFIPS National Computer Conference.

[9]  Bernard P. Zeigler,et al.  Theory of Modelling and Simulation , 1979, IEEE Transactions on Systems, Man and Cybernetics.

[10]  John E. Freund,et al.  Probability and statistics for engineers , 1965 .

[11]  Jack P. C. Kleijnen,et al.  Validation of models: statistical techniques and data availability , 1999, WSC '99.

[12]  Jack P. C. Kleijnen,et al.  EUROPEAN JOURNAL OF OPERATIONAL , 1992 .

[13]  Saul I. Gass,et al.  Feature Article - Decision-Aiding Models: Validation, Assessment, and Related Issues for Policy Analysis , 1983, Oper. Res..

[14]  Robert G. Sargent,et al.  Simulation model verification and validation , 1991, 1991 Winter Simulation Conference Proceedings..

[15]  Osman Balci,et al.  Quality assessment, verification, and validation of modeling and simulation applications , 2004, Proceedings of the 2004 Winter Simulation Conference, 2004..

[16]  Robert G. Sargent Some subjective validation methods using graphical displays of data , 1996, Winter Simulation Conference.

[17]  Peter L. Knepell,et al.  Simulation Validation: A Confidence Assessment Methodology , 1993 .

[19]  Robert E. Shannon Tests for the verification and validation of computer simulation models , 1981, WSC '81.

[20]  Udo W. Pooch,et al.  Integrated Design and Verification of Simulation Programs , 1977, Computer.

[21]  David O. Wood,et al.  MIT model analysis program: what we have learned about policy model review , 1986, WSC '86.

[22]  Connie M. Borror,et al.  Miller and Freund's Probability and Statistics for Engineers, 6th Ed. , 2001 .

[23]  Robert G. Sargent,et al.  Investigation Into Scheduling for an Interactive Computing System , 1974, IBM J. Res. Dev..

[24]  Osman Balci,et al.  Validation of multivariate response models using Hotelling's two-sample T2 test , 1982 .

[25]  Ram Geneshan,et al.  The philosophy of science and validation in simulation , 1993, WSC '93.

[26]  Osman Balci,et al.  Some examples of simulation model validation using hypothesis testing , 1982, WSC.

[27]  R. B. Whitner,et al.  Guidelines for selecting and using simulation model verification techniques , 1989, WSC '89.

[28]  Janina Decker,et al.  Simulation and Model-Based Methodologies: An Integrative View , 1984, NATO ASI Series.

[29]  Robert G. Sargent,et al.  Some approaches and paradigms for verifying and validating simulation models , 2001, Proceeding of the 2001 Winter Simulation Conference (Cat. No.01CH37304).

[30]  J. Kleijnen Statistical tools for simulation practitioners , 1986 .

[31]  Stewart Robinson,et al.  Simulation: The Practice of Model Development and Use , 2004 .

[32]  Thomas H. Naylor,et al.  Verification of Computer Simulation Models , 1967 .

[33]  Osman Balci,et al.  A methodology for cost-risk analysis in the statistical validation of simulation models , 1981, CACM.

[34]  Robert G Sargent,et al.  An Assessment Procedure and a Set of Criteria for Use in the Evaluation of 'Computerized Models and Computer-Based Modelling Tools'. , 1981 .

[35]  Averill M. Law,et al.  Simulation Modeling and Analysis , 1982 .

[36]  Robert G. Sargent,et al.  The use of graphical models in model validation , 1986, WSC '86.

[37]  Robert G. Sargent A tutorial on validation and verification of simulation models , 1988, WSC '88.

[38]  I. A. Chaikovsky,et al.  Terminology for model credibility , 1979 .

[39]  Saul I. Gass,et al.  Model accreditation: A rationale and process for determining a numerical rating , 1993 .

[40]  D. J. Morrice,et al.  FROM SIMULATION TO VERIFICATION ( AND BACK ) , 2003 .

[41]  Robert G. Sargent,et al.  Validation and verification of simulation models , 1999, Proceedings of the 2004 Winter Simulation Conference, 2004..

[42]  Russell C. H. Cheng,et al.  Selecting input models , 1994, Proceedings of Winter Simulation Conference.

[43]  Robert G. Sargent,et al.  A tutorial on verification and validation of simulation models (1984) , 1984, WSC '07.

[44]  M Panjabi,et al.  Validation of mathematical models. , 1979, Journal of biomechanics.

[45]  Averill M. Law,et al.  How to build valid and credible simulation models , 2008, 2008 Winter Simulation Conference.

[46]  G. Enderlein,et al.  Miller, I., and J. E. Freund: Probability and Statistics for Engineers. Prentice‐Hall, Englewood Cliffs, New Jersey 1965. 432 S., Preis 96. s. , 1968 .

[47]  Tuncer Ören,et al.  Concepts and criteria to assess acceptability of simulation studies: a frame of reference , 1981, CACM.

[48]  Gunnar Abrahamson,et al.  Terminology for model credibility , 1980 .

[49]  Robert G. Sargent,et al.  An expository on verification and validation of simulation models , 1985, WSC '85.

[50]  Osman Balci Principles and techniques of simulation validation, verification, and testing , 1995, WSC '95.

[51]  Robert G. Sargent,et al.  Simulation Model Validation , 1984 .

[52]  James D. Arthur,et al.  Expanding our horizons in verification, validation, and accreditation research and practice , 2002, Proceedings of the Winter Simulation Conference.

[53]  Jean-Louis Malouin,et al.  Model validation in operations research , 1983 .

[54]  Edward A. Davis Use of seminar gaming to specify and validate simulation models , 1986, WSC '86.

[55]  J.S. Carson,et al.  Model verification and validation , 2002, Proceedings of the Winter Simulation Conference.

[56]  Osman Balci,et al.  Validation of Multivariate Response Simulation Models by Using Hotelling's Two-sample T^2 Test , 1981 .

[57]  J. Banks,et al.  Discrete-Event System Simulation , 1995 .

[58]  Carl A Mauro,et al.  Factor Screening in Computer Simulation , 1982 .

[59]  Dale K. Pace,et al.  Naval Modeling and Simulation Verification, Validation, and Accreditation , 1993, Proceedings of 1993 Winter Simulation Conference - (WSC '93).

[60]  Saul I. Gass,et al.  Concepts of model confidence , 1981, Comput. Oper. Res..

[61]  Saul I. Gass,et al.  Documenting a Computer-Based Model , 1984 .

[62]  Osman Balci,et al.  Validation fo Multivariate Response Trace-driven Simulation Models , 1983, Performance.

[63]  R. Sargent,et al.  Validation of Simulation Models via Simultaneous Confidence Intervals , 1984 .

[64]  Julie C. Lowery Design of hospital admissions scheduling system using simulation , 1996, Winter Simulation Conference.