History of verification and validation of simulation models

This paper gives the history of verification and validation of discrete-event simulation models as seen through the eyes of its authors and their experiences. The history is divided into three time periods: the early era covering years up to 1970, the awareness era covering the years of the 1970s and 1980s, and the modern era covering the years of 1990 to the present.

[1]  Averill M. Law,et al.  Simulation Modeling and Analysis , 1982 .

[2]  R. Nance,et al.  SOFTWARE REQUIREMENTS ENGINEERING: EXPLORING THE ROLE IN SIMULATION MODEL DEVELOPMENT , 2006 .

[3]  Robert G. Sargent,et al.  The use of graphical models in model validation , 1986, WSC '86.

[4]  Osman Balci,et al.  A methodology for certification of modeling and simulation applications , 2001, TOMC.

[5]  Saul I. Gass,et al.  Model accreditation: A rationale and process for determining a numerical rating , 1993 .

[6]  Udo W. Pooch,et al.  Integrated Design and Verification of Simulation Programs , 1977, Computer.

[7]  Jeffrey P. Buzen,et al.  Computational algorithms for closed queueing networks with exponential servers , 1973, Commun. ACM.

[8]  Osman Balci,et al.  Guidelines For Selecting And Using Simulation Model Verification Techniques , 1989, 1989 Winter Simulation Conference Proceedings.

[9]  I. A. Chaikovsky,et al.  Terminology for model credibility , 1979 .

[10]  Osman Balci,et al.  Validation of multivariate response models using Hotelling's two-sample T2 test , 1982 .

[11]  George B. Kleindorfer,et al.  THE PHILOSOPHY OF SCIENCE AND VAL1DATION IN SIMULATION , 1993 .

[12]  Thomas H. Naylor,et al.  Verification of Computer Simulation Models , 1967 .

[13]  James R. Emshoff,et al.  Design and use of computer simulation models , 1970 .

[14]  Osman Balci,et al.  Some examples of simulation model validation using hypothesis testing , 1982, WSC.

[15]  Bernard P. Zeigler,et al.  Modelling and simulation methodology in the artificial intelligence era. , 1986 .

[16]  R. Sargent,et al.  Validation of Simulation Models via Simultaneous Confidence Intervals , 1984 .

[17]  Saul I. Gass,et al.  Evaluation of complex models , 1977, Comput. Oper. Res..

[18]  Gregory Mahr.  Validation , 2019, Academic Psychiatry.

[19]  A. O'Hagan,et al.  Bayesian calibration of computer models , 2001 .

[20]  George S. Fishman,et al.  On validation of simulation models , 1973, AFIPS National Computer Conference.

[21]  O. Balci.  How to assess the acceptability and credibility of simulation results , 1989, WSC '89.

[22]  Harold G. Hixson.  Equipment maintenance studies using a combination of discrete event and continuous system simulation , 1969 .

[23]  Burton Smith,et al.  A complete interactive simulation environment GPSS/360-NORDEN (1970) , 2007, WSC '07.

[24]  Barry L. Nelson,et al.  Fortieth anniversary special panel: Landmark papers , 2007, 2007 Winter Simulation Conference.

[25]  Robert G. Sargent,et al.  Comparison of two stationary stochastic processes using standardized time series , 1985, WSC '85.

[26]  P. Sanders,et al.  DoD Modeling and Simulation (M&S) Verification, Validation, and Accreditation (VV&A), , 1996 .

[27]  Robert G. Sargent.  Validation of Simulation Models: General Approach , 1992, Concise Encyclopedia of Modelling & Simulation.

[28]  Osman Balci,et al.  Verification, Validation, and Testing , 2007 .

[29]  Richard E. Nance,et al.  Hybrid Simulation: Historical lessons, present challenges and futures , 2016, 2016 Winter Simulation Conference (WSC).

[30]  Bernard P. Zeigler,et al.  Towards a framework for more robust validation and verification of simulation models for systems of systems , 2016 .

[31]  Osman Balci,et al.  Principles of simulation model validation, verification, and testing , 1997 .

[32]  Osman Balci,et al.  A life cycle for modeling and simulation , 2012, Simul..

[33]  Osman Balci,et al.  A methodology for cost-risk analysis in the statistical validation of simulation models , 1981, CACM.

[34]  Robert G Sargent,et al.  An Assessment Procedure and a Set of Criteria for Use in the Evaluation of 'Computerized Models and Computer-Based Modelling Tools'. , 1981 .

[35]  Robert G. Sargent.  Validation of simulation models , 1979, WSC '79.

[36]  James L. McKenney.  Critique of: "Verification of Computer Simulation Models" , 1967 .

[37]  Jr. Harold Albert Anderson.  An empirical investigation into foreground-background scheduling for an interactive computing system , 1972 .

[38]  Y. Barlas.  Multiple tests for validation of system dynamics type of simulation models , 1989 .

[39]  Stewart Robinson,et al.  Simulation verification, validation and confidence: a tutorial , 1999 .

[40]  Osman Balci.  Golden Rules of Verification, Validation, Testing, and Certification of Modeling and Simulation Applications , 2010 .

[41]  Robert G. Sargent,et al.  Investigation Into Scheduling for an Interactive Computing System , 1974, IBM J. Res. Dev..

[42]  David Goldsman,et al.  A tutorial on the operational validation of simulation models , 2016, 2016 Winter Simulation Conference (WSC).

[43]  Amy Henninger,et al.  VV&A of M&S in the DoD: Moving from Art to Science , 2010 .

[44]  Thomas H. Naylor,et al.  Design of computer simulation experiments for industrial systems , 1966, CACM.

[45]  J. D. Johannes,et al.  Systems Simulation: The Art and Science , 1975, IEEE Transactions on Systems, Man, and Cybernetics.

[46]  Lee W. Schruben,et al.  Establishing the credibility of simulations , 1980 .

[47]  Paul K. Davis,et al.  Generalizing Concepts and Methods of Verification, Validation, and Accreditation (VV&A) for Military Simulations, , 1992 .

[48]  James D. Arthur,et al.  Expanding our horizons in verification, validation, and accreditation research and practice , 2002, Proceedings of the Winter Simulation Conference.

[49]  James D. Arthur,et al.  Improving the model development process: expanding our horizons in verification, validation, and accreditation research and practice , 2002, WSC '02.

[50]  Jack P. C. Kleijnen,et al.  A methodology for fitting and validating metamodels in simulation , 2000, Eur. J. Oper. Res..

[51]  Robert G. Sargent,et al.  Confidence interval estimation for the variance parameter of stationary , 1990 .

[52]  Saul I. Gass,et al.  Concepts of model confidence , 1981, Comput. Oper. Res..

[53]  Yuan Guo-xing,et al.  Verification and Validation in Scientific Computing Code , 2010 .

[54]  Richard E. Nance,et al.  The Simulation Project Life-Cycle: Models and Realities , 2006, Proceedings of the 2006 Winter Simulation Conference.

[55]  Robert G. Sargent,et al.  A tutorial on verification and validation of simulation models (1984) , 1984, WSC '07.

[56]  Thomas Herbert Naylor.  Computer Simulation Techniques , 1966 .

[57]  Osman Balci,et al.  Verification, validation, and accreditation , 1998, 1998 Winter Simulation Conference. Proceedings (Cat. No.98CH36274).

[58]  Osman Balci,et al.  Validation fo Multivariate Response Trace-driven Simulation Models , 1983, Performance.

[59]  Stewart Robinson,et al.  Independent Verification and Validation of an Industrial Simulation Model , 2010, Simul..

[60]  Yaman Barlas,et al.  Formal aspects of model validity and validation in system dynamics , 1996 .

[61]  Saul I. Gass.  Validation and assessment of energy models , 1981 .

[62]  Robert E. Shannon.  Tests for the verification and validation of computer simulation models , 1981, WSC '81.

[63]  James R. Wilson,et al.  A brief history of simulation revisited , 2010, Proceedings of the 2010 Winter Simulation Conference.

[64]  Averill M. Law,et al.  How to build valid and credible simulation models , 2008, 2008 Winter Simulation Conference.

[65]  Saul I. Gass,et al.  An Assessment Procedure for Simulation Models: A Case Study , 1991, Oper. Res..

[66]  William L. Maxwell,et al.  Some Problems of Digital Systems Simulation , 1959 .

[67]  Robert G. Sargent,et al.  An interval statistical procedure for use in validation of simulation models , 2015, J. Simulation.

[68]  Gunnar Abrahamson,et al.  Terminology for model credibility , 1980 .

[69]  Saul I. Gass,et al.  Letter to the Editor - Guidelines for Model Evaluation: An Abridged Version of the U.S. General Accounting Office Exposure Draft , 1980, Oper. Res..

[70]  Bernard P. Zeigler,et al.  Theory of Modelling and Simulation , 1979, IEEE Transactions on Systems, Man, and Cybernetics.

[71]  Jack P. C. Kleijnen,et al.  EUROPEAN JOURNAL OF OPERATIONAL , 1992 .

[72]  P. E. Barker,et al.  Calibrating the simulation model of the IBM system/360 time sharing system , 1969 .

[73]  Osman Balci.  Credibility assessment of simulation results , 1986, WSC '86.

[74]  James E. Coolahan,et al.  Simulation Verification, Validation, and Accreditation , 2000 .

[75]  Osman Balci,et al.  Quality assessment, verification, and validation of modeling and simulation applications , 2004, Proceedings of the 2004 Winter Simulation Conference, 2004..

[76]  Richard L. Van Horn,et al.  Validation of Simulation Results , 1971 .

[77]  Robert G. Sargent.  Some subjective validation methods using graphical displays of data , 1996, Winter Simulation Conference.

[78]  Saul I. Gass,et al.  Feature Article - Decision-Aiding Models: Validation, Assessment, and Related Issues for Policy Analysis , 1983, Oper. Res..

[79]  Osman Balci.  Validation, verification, and testing techniques throughout the life cycle of a simulation study , 1994, WSC '94.

[80]  Peter L. Knepell,et al.  Simulation Validation: A Confidence Assessment Methodology , 1993 .

[81]  Robert G. Sargent.  Model Verification and Validation , 2015 .

[82]  Charles Gilbert Moore,et al.  Network models for large-scale time-sharing systems , 1971 .

[83]  Bernard P. Zeigler,et al.  Multifacetted Modelling and Discrete Event Simulation , 1984 .

[84]  Mansooreh Mollaghasemi,et al.  Validation and verification of the simulation model of a photolithography process in semiconductor manufacturing , 1998, 1998 Winter Simulation Conference. Proceedings (Cat. No.98CH36274).

[85]  C. West Churchman,et al.  Introduction to operations research , 1957 .

[86]  Osman Balci,et al.  A collaborative evaluation environment for credibility assessment of modeling and simulation applications , 2002, Proceedings of the Winter Simulation Conference.

[87]  George S. Fishman,et al.  The statistics of discrete-event simulation , 1968 .

[88]  J. Banks.  Testing, understanding, and validating complex simulation models , 1989, WSC '89.

[89]  M. P. Fabisch,et al.  Safeguard data-processing system: Introduction and overview , 1975, The Bell System Technical Journal.

[90]  David Goldsman,et al.  Use of the interval statistical procedure for simulation model validation , 2015, 2015 Winter Simulation Conference (WSC).

[91]  H. G. Natke.  Model verification and validation in enginnering , 1994 .

[92]  J. Kleijnen,et al.  Validation of models: statistical techniques and data availability , 1999, WSC'99. 1999 Winter Simulation Conference Proceedings. 'Simulation - A Bridge to the Future' (Cat. No.99CH37038).

[93]  Mohammad Jamshidi,et al.  System of systems engineering : innovations for the 21st century , 2008 .

[94]  Robert G. Sargent,et al.  Using Standardized Time Series to Estimate Confidence Intervals for the Difference Between Two Stationary Stochastic Processes , 1987, Oper. Res..

[95]  Robert G. Sargent,et al.  Some approaches and paradigms for verifying and validating simulation models , 2001, Proceeding of the 2001 Winter Simulation Conference (Cat. No.01CH37304).