EUROPEAN JOURNAL OF OPERATIONAL

This paper surveys verification and validation of models, especially simulation models in operations research. For verification it discusses 1) general good programming practice (such as modular programming), 2) checking intermediate simulation outputs through tracing and statistical testing per module, 3) statistical testing of final simulation outputs against analytical results, and 4) animation. For validation it discusses 1) obtaining real-worl data, 2) comparing simulated and real data through simple tests such as graphical, Schruben-Turing, and t tests, 3) testing whether simulated and real responses are positively correlated and moreover have the same mean, using two new statistical procedures based on regression analysis, 4) sensitivity analysis based on design of experiments and regression analysis, and risk or uncertainty analysis based on Monte Carlo sampling, and 5) white versus black box simulation models. Both verification and validation require good documentation, and are crucial parts of assessment, credibility, and accreditation. A bibliography with 61 references is included.

[1]  Saul I. Gass,et al.  Documenting a Computer-Based Model , 1984 .

[2]  John S. Carson Verification And Validation: A Consultant's Perspective , 1989, 1989 Winter Simulation Conference Proceedings.

[3]  F. David,et al.  Statistical Techniques in Simulation: Part I , 1975 .

[4]  James S. Hodges,et al.  Six (Or So) Things You Can Do with a Bad Model , 1991, Oper. Res..

[5]  Lee W. Schruben,et al.  Establishing the credibility of simulations , 1980 .

[6]  Yaman Barlas,et al.  Philosophical roots of model validation: Two paradigms , 1990 .

[7]  Paul K. Davis,et al.  Generalizing Concepts and Methods of Verification, Validation, and Accreditation (VV&A) for Military Simulations, , 1992 .

[8]  Jack P. C. Kleijnen,et al.  Simulation: A Statistical Perspective , 1992 .

[9]  Clive W. J. Granger,et al.  Pitfalls of Analysis , 1981 .

[10]  F. V. Krumm,et al.  Management and Application of Decision and Risk Analysis in Du Pont , 1992 .

[11]  Ted Eschenbach,et al.  Spiderplots versus Tornado Diagrams for Sensitivity Analysis , 1992 .

[12]  Muhittin Oral,et al.  In search of a valid view of model validation for operations research , 1993 .

[13]  Heston R. Hicks,et al.  Comparison of models: ex post facto validation/acceptance? , 1992, WSC '92.

[14]  Jack P. C. Kleijnen,et al.  Techniques for sensitivity analysis of simulation models: A case study of the CO2 greenhouse effect , 1992, Simul..

[15]  R. Sargent,et al.  Validation of Simulation Models via Simultaneous Confidence Intervals , 1984 .

[16]  D. J. Murray-Smith Problems and prospects in the validation of dynamic models , 1992 .

[17]  Nicholas V. Findler,et al.  A system for automatic model verification and validation , 1990 .

[18]  R. Deal Simulation Modeling and Analysis (2nd Ed.) , 1994 .

[19]  Walter J. Karplus,et al.  The Spectrum of Mathematical Models. , 1983 .

[20]  Harold Stanislaw,et al.  Tests of Computer Simulation Validity , 1986 .

[21]  John D. C. Little,et al.  Operations Research in Industry: New Opportunities in a Changing World , 1991, Oper. Res..

[22]  D. Murray-Smith,et al.  The external validation of nonlinear models for helicopter dynamics , 1993 .

[23]  C. D. Beaumont Pitfalls of Analysis , 1981 .

[24]  Randall P. Sadowski,et al.  Introduction to Simulation Using Siman , 1990 .

[25]  Jack P. C. Kleijnen,et al.  Statistics and deterministic simulation models: why not? , 1990, 1990 Winter Simulation Conference Proceedings.

[26]  B. D. Ripley,et al.  Uses and abuses of statistical simulation , 1988, Math. Program..

[27]  Hugh J. Miser,et al.  A foundational concept of science appropriate for validation in operational research , 1993 .

[28]  Paul K. Davis,et al.  An Introduction to Variable-Resolution Modeling and Cross-Resolution Model Connection , 1993 .

[29]  John F. Passafiume,et al.  Software testing and evaluation , 1987 .

[30]  K. D. Tocher,et al.  The art of simulation , 1967 .

[31]  Jack P. C. Kleijnen,et al.  Sensitivity analysis versus uncertainty analysis: when to use what? in predictability and nonlinear , 1994 .

[32]  J. Kleijnen Computers and Profits: Quantifying Financial Benefits of Information , 1980 .

[33]  Averill M. Law,et al.  Simulation Modeling and Analysis , 1982 .

[34]  M. B. Beck,et al.  Water quality modeling: A review of the analysis of uncertainty , 1987 .

[35]  John A. Deacon Intelligent Vehicle Highway System , 1989 .

[36]  J. Kleijnen Statistical tools for simulation practitioners , 1986 .

[37]  Richard A Fleming,et al.  Evaluating Models for Spruce Budworm-Forest Management: Comparing Output with Regional Field Data. , 1992, Ecological applications : a publication of the Ecological Society of America.

[38]  Robert Laurence Baber The spine of software - designing provably correct software: theory and practice: or a mathematical introduction to the semantics of computer programs , 1986 .

[39]  John C. Cherniavsky,et al.  Validation, Verification, and Testing of Computer Software , 1982, CSUR.

[40]  Saul I. Gass,et al.  Model accreditation: A rationale and process for determining a numerical rating , 1993 .

[41]  Peter L. Knepell,et al.  Simulation Validation: A Confidence Assessment Methodology , 1993 .

[42]  Enver Yücesan,et al.  Proceedings of the 1989 winter simulation conference , 1989 .

[43]  Brian Unger WSC'90: An after-conference report , 1991, Simul..

[44]  Jack P. C. Kleijnen,et al.  Simulation and optimization in production planning: A case study , 1993, Decis. Support Syst..

[45]  Jack P. C. Kleijnen,et al.  Statistical Techniques in Simulation , 1977, IEEE Transactions on Systems, Man and Cybernetics.

[46]  Y. Barlas Multiple tests for validation of system dynamics type of simulation models , 1989 .

[47]  Arthur M. Geoffrion,et al.  Forces, Trends, and Opportunities in MS/OR , 1992, Oper. Res..

[48]  Bernard P. Zeigler,et al.  Theory of Modelling and Simulation , 1979, IEEE Transactions on Systems, Man and Cybernetics.

[49]  Osman Balci,et al.  Guidelines For Selecting And Using Simulation Model Verification Techniques , 1989, 1989 Winter Simulation Conference Proceedings.

[50]  Yaman Barlas,et al.  An autocorrelation function test for out put validation , 1990, Simul..

[51]  Bernard P. Zeigler,et al.  Theory of Modelling and Simulation , 1979, IEEE Transactions on Systems, Man, and Cybernetics.

[52]  Shiv K. Gupta Pitfalls of analysis: Giandomenico MAJONE and Edward S. QUADE (Eds.) Wiley, Chichester, 1980, 213 pages, £ 12.50 , 1981 .

[53]  Samuel E. Bodily,et al.  Introduction: The Practice of Decision and Risk Analysis , 1992 .

[54]  Jack P. C. Kleijnen,et al.  Identifying the important factors in simulation models with many factors , 1991 .

[55]  Thomas J. Schriber,et al.  An introduction to simulation using GPSS/H , 1991 .

[56]  Steve Bankes,et al.  Exploratory Modeling for Policy Analysis , 1993, Oper. Res..

[57]  Thomas Herbert Naylor Computer Simulation Techniques , 1966 .

[58]  Saul I. Gass,et al.  An Assessment Procedure for Simulation Models: A Case Study , 1991, Oper. Res..

[59]  J.P.C. Kleijnen,et al.  Validation of simulation models : Mine-hunting case-study , 1992 .

[60]  J. Banks Testing, understanding, and validating complex simulation models , 1989, WSC '89.

[61]  Robert G. Sargent,et al.  Simulation model verification and validation , 1991, 1991 Winter Simulation Conference Proceedings..

[62]  George P. Richardson,et al.  Introduction to System Dynamics Modeling with DYNAMO , 1981 .

[63]  J. D. Johannes,et al.  Systems Simulation: The Art and Science , 1975, IEEE Transactions on Systems, Man, and Cybernetics.

[64]  Izak Benbasat,et al.  A framework for the validation of knowledge acquisition , 1989 .

[65]  Ole-Johan Dahl Verifiable programming , 1992, Prentice Hall international series in computer science.

[66]  Jack P. C. Kleijnen,et al.  Statistical Techniques in Simulation , 1977, IEEE Transactions on Systems, Man and Cybernetics.

[67]  Muhittin Oral,et al.  The facets of the modeling and validation process in operations research , 1993 .

[68]  Adrian Pagan,et al.  On the role of simulation in the statistical evaluation of econometric models , 1989 .