Verification, validation and accreditation of simulation models

The paper discusses verification, validation, and accreditation of simulation models. The different approaches to deciding model validity are presented; how model verification and validation relate to the model development process are discussed; various validation techniques are defined; conceptual model validity, model verification, operational validity, and data validity are described; ways to document results are given; a recommended procedure is presented; and accreditation is briefly discussed.

[1]  Thomas H. Naylor,et al.  Verification of Computer Simulation Models , 1967 .

[2]  G. Enderlein,et al.  Miller, I., and J. E. Freund: Probability and Statistics for Engineers. Prentice‐Hall, Englewood Cliffs, New Jersey 1965. 432 S., Preis 96. s. , 1968 .

[3]  Robert G. Sargent,et al.  Investigation Into Scheduling for an Interactive Computing System , 1974, IBM J. Res. Dev..

[4]  J. D. Johannes,et al.  Systems Simulation: The Art and Science , 1975, IEEE Transactions on Systems, Man, and Cybernetics.

[5]  Bernard P. Zeigler,et al.  Theory of Modelling and Simulation , 1979, IEEE Transactions on Systems, Man and Cybernetics.

[6]  Bernard P. Zeigler,et al.  Theory of Modelling and Simulation , 1979, IEEE Transactions on Systems, Man, and Cybernetics.

[7]  Robert G. Sargent.  Validation of simulation models , 1979, WSC '79.

[8]  M Panjabi,et al.  Validation of mathematical models. , 1979, Journal of biomechanics.

[9]  Gunnar Abrahamson,et al.  Terminology for model credibility , 1980 .

[10]  Lee W. Schruben,et al.  Establishing the credibility of simulations , 1980 .

[11]  Saul I. Gass,et al.  Letter to the Editor - Guidelines for Model Evaluation: An Abridged Version of the U.S. General Accounting Office Exposure Draft , 1980, Oper. Res..

[12]  Osman Balci,et al.  A methodology for cost-risk analysis in the statistical validation of simulation models , 1981, CACM.

[13]  Robert G Sargent,et al.  An Assessment Procedure and a Set of Criteria for Use in the Evaluation of 'Computerized Models and Computer-Based Modelling Tools'. , 1981 .

[14]  Osman Balci,et al.  Validation of Multivariate Response Simulation Models by Using Hotelling's Two-sample T^2 Test , 1981 .

[15]  Saul I. Gass,et al.  Concepts of model confidence , 1981, Comput. Oper. Res..

[16]  Tuncer Ören,et al.  Concepts and criteria to assess acceptability of simulation studies: a frame of reference , 1981, CACM.

[17]  Osman Balci,et al.  Some examples of simulation model validation using hypothesis testing , 1982, WSC.

[18]  Averill M. Law,et al.  Simulation Modeling and Analysis , 1982 .

[19]  Osman Balci,et al.  Validation of multivariate response models using Hotelling's two-sample T2 test , 1982 .

[20]  Saul I. Gass,et al.  Feature Article - Decision-Aiding Models: Validation, Assessment, and Related Issues for Policy Analysis , 1983, Oper. Res..

[21]  Osman Balci,et al.  Validation fo Multivariate Response Trace-driven Simulation Models , 1983, Performance.

[22]  Robert G. Sargent,et al.  Simulation Model Validation , 1984 .

[23]  R. Sargent,et al.  Validation of Simulation Models via Simultaneous Confidence Intervals , 1984 .

[24]  Saul I. Gass,et al.  Documenting a Computer-Based Model , 1984 .

[25]  Robert G. Sargent,et al.  An expository on verification and validation of simulation models , 1985, WSC '85.

[26]  David O. Wood,et al.  MIT model analysis program: what we have learned about policy model review , 1986, WSC '86.

[27]  Robert G. Sargent,et al.  The use of graphical models in model validation , 1986, WSC '86.

[28]  Gideon Oron.  Statistical tools for simulation practitioners , 1988 .

[29]  Robert G. Sargent.  A tutorial on validation and verification of simulation models , 1988, WSC '88.

[30]  Osman Balci,et al.  Guidelines For Selecting And Using Simulation Model Verification Techniques , 1989, 1989 Winter Simulation Conference Proceedings.

[31]  O. Balci.  How to assess the acceptability and credibility of simulation results , 1989, WSC '89.

[32]  Robert G. Sargent,et al.  Simulation model verification and validation , 1991, 1991 Winter Simulation Conference Proceedings..

[33]  Jack P. C. Kleijnen,et al.  EUROPEAN JOURNAL OF OPERATIONAL , 1992 .

[34]  Peter L. Knepell,et al.  Simulation Validation: A Confidence Assessment Methodology , 1993 .

[35]  George B. Kleindorfer,et al.  THE PHILOSOPHY OF SCIENCE AND VAL1DATION IN SIMULATION , 1993 .

[36]  Richard A. Johnson.  Miller & Freund's Probability and Statistics for Engineers , 1993 .

[37]  Saul I. Gass,et al.  Model accreditation: A rationale and process for determining a numerical rating , 1993 .

[38]  Ram Geneshan,et al.  The philosophy of science and validation in simulation , 1993, WSC '93.

[39]  R. Deal.  Simulation Modeling and Analysis (2nd Ed.) , 1994 .

[40]  J. Banks,et al.  Discrete-Event System Simulation , 1995 .

[41]  Robert G. Sargent.  Some subjective validation methods using graphical displays of data , 1996, Winter Simulation Conference.

[42]  Osman Balci,et al.  Verification, validation, and accreditation , 1998, 1998 Winter Simulation Conference. Proceedings (Cat. No.98CH36274).

[43]  Jack P. C. Kleijnen,et al.  Validation of models: statistical techniques and data availability , 1999, WSC '99.

[44]  I. Oren,et al.  Concepts and Criteria to Assess Acceptability of Simulation Studies : A Frame of Reference , 2001 .