Verification, Validation, and Predictive Capability in Computational Engineering and Physics

Developers of computer codes, analysts who use the codes, and decision makers who rely on the results of the analyses face a critical question: How should confidence in modeling and simulation be critically assessed? Verification and validation (V&V) of computational simulations are the primary methods for building and quantifying this confidence. Briefly, verification is the assessment of the accuracy of the solution to a computational model. Validation is the assessment of the accuracy of a computational simulation by comparison with experimental data. In verification, the relationship of the simulation to the real world is not an issue. In validation, the relationship between computation and the real world, i.e., experimental data, is the issue.

[1]  Margaret J. Robertson,et al.  Design and Analysis of Experiments , 2006, Handbook of statistics.

[2]  I NICOLETTI,et al.  The Planning of Experiments , 1936, Rivista di clinica pediatrica.

[3]  H. Keller,et al.  Accurate Difference Methods for Linear Ordinary Differential Systems Subject to Linear Constraints , 1969 .

[4]  I. A. Chaikovsky,et al.  Terminology for model credibility , 1979 .

[5]  B. N. Srivastava,et al.  A finite difference technique involving discontinuous derivatives , 1979 .

[6]  Gunnar Abrahamson,et al.  Terminology for model credibility , 1980 .

[7]  F. G. Blottner,et al.  Influence of boundary approximations and conditions on finite-difference solutions , 1982 .

[8]  Averill M. Law,et al.  Simulation Modeling and Analysis , 1982 .

[9]  Patrick J. Roache,et al.  Symbolic manipulation and computational fluid dynamics , 1983 .

[10]  J. Whiteman.  The Mathematics of Finite Elements and Applications. , 1983 .

[11]  R. Lowen,et al.  On the fundamentals of fuzzy sets. , 1984 .

[12]  Frank Jay,et al.  IEEE standard dictionary of electrical and electronics terms , 1984 .

[13]  Richard E. Fairley,et al.  Software engineering concepts , 1985, McGraw-Hill series in software engineering and technology.

[14]  T. M. Shih,et al.  A procedure to debug computer programs , 1985 .

[15]  Robert Laurence Baber.  The spine of software - designing provably correct software: theory and practice: or a mathematical introduction to the semantics of computer programs , 1986 .

[16]  Eli Turkel,et al.  Accuracy of schemes with nonuniform meshes for compressible fluid flows , 1986 .

[17]  J. Kleijnen.  Statistical tools for simulation practitioners , 1986 .

[18]  Robert E. Melchers,et al.  Structural Reliability: Analysis and Prediction , 1987 .

[19]  John F. Passafiume,et al.  Software testing and evaluation , 1987 .

[20]  M. B. Beck,et al.  Water quality modeling: A review of the analysis of uncertainty , 1987 .

[21]  Ivo Babuška,et al.  Pollution problem of the p‐ and h–p versions of the finite element method , 1987 .

[22]  C. Hirsch,et al.  Numerical Computation of Internal and External Flows. By C. HIRSCH. Wiley. Vol. 1, Fundamentals of Numerical Discretization. 1988. 515 pp. £60. Vol. 2, Computational Methods for Inviscid and Viscous Flows. 1990, 691 pp. £65. , 1991, Journal of Fluid Mechanics.

[23]  Charles Hirsch,et al.  Numerical computation of internal and external flows (vol1: Fundamentals of numerical discretization) , 1991 .

[24]  Charles Hirsch,et al.  Numerical computation of internal & external flows: fundamentals of numerical discretization , 1988 .

[25]  V. Rich.  Personal communication , 1989, Nature.

[26]  T. Shih,et al.  Effects of grid staggering on numerical schemes , 1989 .

[27]  Hugh W. Coleman,et al.  Experimentation and Uncertainty Analysis for Engineers , 1989 .

[28]  C. Hirsch.  Numerical Computation of Internal and External Flows, Volume 2: Computational Methods for Inviscid and Viscous Flows , 1990 .

[29]  Paul Rook,et al.  Software Reliability Handbook , 1990 .

[30]  I. Catton,et al.  Quantifying reactor safety margins part 1: An overview of the code scaling, applicability, and uncertainty evaluation methodology , 1990 .

[31]  B. E. Boyack,et al.  Quantifying reactor safety margins part 2: Characterization of important contributors to uncertainty , 1990 .

[32]  C. Hirsch.  Computational methods for inviscid and viscous flows , 1990 .

[33]  B. E. Boyack,et al.  Quantifying reactor safety margins part 3: Assessment and ranging of parameters , 1990 .

[34]  Helmut A. Partsch,et al.  Specification and transformation of programs , 1990 .

[35]  P. Roache.  Need for Control of Numerical Accuracy , 1990 .

[36]  William L. Oberkampf,et al.  Joint Computational/Experimental Aerodynamics Research on a Hypersonic Vehicle, Part 1: Experimental Results , 1991 .

[37]  William L. Oberkampf,et al.  Joint computational/experimental aerodynamics research on hypersonicvehicle. II - Computational results , 1991 .

[38]  Nissim Francez,et al.  Program verification , 1992, International computer science series.

[39]  Michael Dyer.  The Cleanroom Approach to Quality Software Development , 1992, Int. CMG Conference.

[40]  Ole-Johan Dahl.  Verifiable programming , 1992, Prentice Hall international series in computer science.

[41]  Jack P. C. Kleijnen,et al.  EUROPEAN JOURNAL OF OPERATIONAL , 1992 .

[42]  Peter L. Knepell,et al.  Simulation Validation: A Confidence Assessment Methodology , 1993 .

[43]  William L. Oberkampf,et al.  Experimental aerodynamics research on a hypersonic vehicle , 1993 .

[44]  Jon C. Helton,et al.  Uncertainty and sensitivity analysis techniques for use in performance assessment for radioactive waste disposal , 1993 .

[45]  R. D. Smet.  Simulation modeling and analysis (2nd edition): Averill M. Law and W. David Kelton McGraw-Hill, Inc., New York, 1991, xxii + 759 pages, £31.05, ISBN 0 07 036698 5 , 1993 .

[46]  C. Julius Wang,et al.  What Every Engineer Should Know about Reliability and Risk Analysis , 1993 .

[47]  Saul I. Gass,et al.  Model accreditation: A rationale and process for determining a numerical rating , 1993 .

[48]  James R. Matey,et al.  How Statistical Design Concepts Can Improve Experimentation in the Physical Sciences , 1993 .

[49]  William Oberkampf.  A proposed framework for computational fluid dynamics code calibration and validation , 1993 .

[50]  Daniel C. Haworth,et al.  A global approach to error estimation and physical diagnostics in multidimensional computational fluid dynamics , 1993 .

[51]  P. Roache.  Perspective: A Method for Uniform Reporting of Grid Refinement Studies , 1994 .

[52]  Patrick J. Roache.  Response: To the Comments by Drs. W. Shyy and M. Sindir , 1994 .

[53]  M. Jenner.  The cleanroom approach to quality software development : M Dyer John Wiley (1992) 198 pp £42.50 ISBN 0 471 54823 5 , 1994, Inf. Softw. Technol..

[54]  Hartmut Bossel,et al.  Modeling and simulation , 1994 .

[55]  Bruce Elliott,et al.  Testing Computer Software , 1994 .

[56]  Peter Kafka.  Important issues using PSA technology for design of new systems and plants , 1994 .

[57]  R. Deal.  Simulation Modeling and Analysis (2nd Ed.) , 1994 .

[58]  Kameshwar Poolla,et al.  Statistical validation for uncertainty models , 1995 .

[59]  T. Hamill,et al.  A Probabilistic Forecast Contest and the Difficulty in Assessing Short-Range Forecast Uncertainty , 1995 .

[60]  Jack P. C. Kleijnen,et al.  Statistical validation of simulation models , 1995 .

[61]  David Draper,et al.  Assessment and Propagation of Model Uncertainty , 2011 .

[62]  Ivo Babuška,et al.  A posteriori estimation and adaptive control of the pollution error in the h‐version of the finite element method , 1995 .

[63]  S. B. Kiselev,et al.  The capability maturity model: guidelines for improving the software process , 1995 .

[64]  J. Kohlas,et al.  A Mathematical Theory of Hints: An Approach to the Dempster-Shafer Theory of Evidence , 1995 .

[65]  Jonathan P. Bowen,et al.  Applications of Formal Methods , 1995 .

[66]  William L. Oberkampf,et al.  Surface pressure measurements for CFD code validation in hypersonic flow , 1995 .

[67]  Jonathan P. Bowen,et al.  Ten Commandments of Formal Methods , 1995, Computer.

[68]  William L. Oberkampf,et al.  A proposed methodology for computational fluid dynamics code verification, calibration, and validation , 1995, ICIASF '95 Record. International Congress on Instrumentation in Aerospace Simulation Facilities.

[69]  Jürg Kohlas,et al.  A Mathematical Theory of Hints , 1995 .

[70]  Joseph G. Marvin,et al.  Perspective on Computational Fluid Dynamics Validation , 1995 .

[71]  Raymond Cosner.  CFD validation requirements for technology transition , 1995 .

[72]  Russell G. Almond.  Graphical belief modeling , 1995 .

[73]  Stephen C. Hora,et al.  Aleatory and epistemic uncertainty in probability elicitation with an example from hazardous waste management , 1996 .

[74]  Les Hatton,et al.  The T-experiments: errors in scientific software , 1996, Quality of Numerical Software.

[75]  David M. Johnson,et al.  A review of fault management techniques used in safety-critical avionic systems , 1996 .

[76]  Hiromitsu Kumamoto,et al.  Probabilistic Risk Assessment and Management for Engineers and Scientists , 1996 .

[77]  K. Poolla,et al.  On Statistical Model Validation , 1996 .

[78]  M. Elisabeth Paté-Cornell,et al.  Uncertainties in risk analysis: Six levels of treatment , 1996 .

[79]  Kathryn B. Laskey.  Model uncertainty: theory and practical implications , 1996, IEEE Trans. Syst. Man Cybern. Part A.

[80]  M. G. Marietta,et al.  Uncertainty and sensitivity analysis results obtained in the 1992 performance assessment for the waste isolation pilot plant , 1996 .

[81]  S. Ferson,et al.  Different methods are needed to propagate ignorance and variability , 1996 .

[82]  D. S. Sivia,et al.  Data Analysis , 1996, Encyclopedia of Evolutionary Psychological Science.

[83]  Dolores R. Wallace,et al.  Reference information for the software verification and validation process , 1996 .

[84]  Henrik O. Madsen,et al.  Structural Reliability Methods , 1996 .

[85]  Daniel Chan,et al.  On the development and demonstration of a code validation process for industrial applications , 1996 .

[86]  E. D. Lynch,et al.  Overview of the state-of-the-practice of computational fluid dynamics in advanced propulsion system design , 1997 .

[87]  Jun Du,et al.  Short-Range Ensemble Forecasting of Quantitative Precipitation , 1997 .

[88]  William L. Oberkampf,et al.  Experimental methodology for computational fluid dynamics code validation , 1997 .

[89]  John D. Denton,et al.  Lessons from rotor 37 , 1997 .

[90]  Osman Balci,et al.  Principles of simulation model validation, verification, and testing , 1997 .

[91]  J. C. Helton,et al.  Uncertainty and sensitivity analysis in the presence of stochastic and subjective uncertainty , 1997 .

[92]  William L. Oberkampf,et al.  Uncertainty and error in computational simulations , 1997 .

[93]  Hugh W. Coleman,et al.  Uncertainties and CFD Code Validation , 1997 .

[94]  Ivo Babuška,et al.  A posteriori error estimation for finite element solutions of Helmholtz' equation—Part II: estimation of the pollution error , 1997 .

[95]  Kiran Alapaty,et al.  UNCERTAINTY IN THE SPECIFICATION OF SURFACE CHARACTERISTICS: A STUDY OF PREDICTION ERRORS IN THE BOUNDARY LAYER , 1997 .

[96]  A J Chorin,et al.  Optimal prediction of underresolved dynamics. , 1998, Proceedings of the National Academy of Sciences of the United States of America.

[97]  Culbert B. Laney,et al.  Computational Gasdynamics: Waves , 1998 .

[98]  Patrick J. Roache,et al.  Verification and Validation in Computational Science and Engineering , 1998 .

[99]  P. Roache.  Verification of Codes and Calculations , 1998 .

[100]  George B. Kleindorfer,et al.  Validation in Simulation: Various Positions in the Philosophy of Science , 1998 .

[101]  Yusheng Feng,et al.  LOCAL AND POLLUTION ERROR ESTIMATION FOR STOKESIAN FLOWS , 1998 .

[102]  B. E. Boyack,et al.  An integrated structure and scaling methodology for severe accident technical issue resolution: Development of methodology , 1998 .

[103]  G. E. Wilson,et al.  The role of the PIRT process in experiments, code development and code applications associated with Reactor Safety analysis , 1998 .

[104]  William L. Oberkampf,et al.  Guide for the verification and validation of computational fluid dynamics simulations , 1998 .

[105]  William L. Oberkampf,et al.  Issues in Computational Fluid Dynamics Code Verification and Validation , 1997 .

[106]  Luigi Martinelli,et al.  Mesh Refinement and Modeling Errors in Flow Simulation , 1998 .

[107]  D. J. Murraysmith,et al.  Methods for the external validation of contiuous system simulation models:a review , 1998 .

[108]  John R. Tucker.  Statistical Software Engineering , 1998 .

[109]  AshokK . Singhal.  KEY ELEMENTS OF VERIFICATION AND VALIDATION OF CFD SOFTWARE , 1998 .

[110]  G. Klir,et al.  Uncertainty-based information: Elements of generalized information theory (studies in fuzziness and soft computing). , 1998 .

[111]  H. C. Yee,et al.  Aspects of Numerical Uncertainties in Time Marching to Steady-State Numerical Solutions , 1998 .

[112]  John Gustafson,et al.  Computational Verifiability and Feasibilitv J of the ASCI Program , 1998 .

[113]  Alexandre J. Chorin,et al.  On the prediction of large-scale dynamics using unresolved computations , 1998 .

[114]  David H. Sharp,et al.  Stochastic methods for the prediction of complex multiscale phenomena , 1998 .

[115]  Timothy G. Trucano,et al.  Statistical Validation of Engineering and Scientific Models: Background , 1999 .

[116]  Kenneth F. Alvin,et al.  A New Methodology for the Estimation of Total Uncertainty in Computational Simulation , 1999 .

[117]  Yachen Lin.  Introductory Statistics and Random Phenomena , 1999, Technometrics.

[118]  Achintya Haldar,et al.  Probability, Reliability and Statistical Methods in Engineering Design (Haldar, Mahadevan) , 1999 .

[119]  TIMOTHY G. TRUCANO,et al.  Guidelines for Sandia ASCI Verification and Validation Plans - Content and Format: Version 1.0 , 1999 .

[120]  Alexandre J. Chorin,et al.  Unresolved Computation and Optimal Predictions , 1999 .

[121]  Kenneth M. Hanson,et al.  A framework for assessing uncertainties in simulation predictions , 1999 .

[122]  George J. Klir,et al.  Uncertainty-Based Information , 1999 .

[123]  D. E. Stevenson.  A critical look at quality in large-scale simulations , 1999, Comput. Sci. Eng..

[124]  Tony L. Wahl,et al.  Experimentation and Uncertainty Analysis for Engineers, 2nd Ed. , 1999 .

[125]  Stewart Robinson,et al.  Simulation verification, validation and confidence: a tutorial , 1999 .

[126]  M. Carpenter,et al.  Accuracy of Shock Capturing in Two Spatial Dimensions , 1999 .

[127]  J. C. Helton,et al.  Uncertainty and sensitivity analysis in performance assessment for the Waste Isolation Pilot Plant , 1999 .

[128]  M. V. Frank.  Treatment of uncertainties in space nuclear risk assessment with examples from Cassini mission applications , 1999 .

[129]  Francisco J. Samaniego,et al.  The Practice of Bayesian Analysis , 1999, Technometrics.

[130]  Shin Ta Liu,et al.  Risk Modeling, Assessment, and Management , 1999, Technometrics.

[131]  Simon P. Wilson,et al.  Statistical methods in software engineering : reliability and risk , 1999 .

[132]  H. Christopher Frey,et al.  Probabilistic Techniques in Exposure Assessment: A Handbook for Dealing with Variability and Uncertainty in Models and Inputs , 1999 .

[133]  M. Elisabeth Paté-Cornell.  Conditional Uncertainty Analysis and Implications for Decision Making: The Case of WIPP , 1999, Risk analysis : an official publication of the Society for Risk Analysis.

[134]  J. Chilès,et al.  Geostatistics: Modeling Spatial Uncertainty , 1999 .

[135]  J. Glimm,et al.  A Probability Model For Errors in the Numerical Solutions of a Partial Differential Equation , 1999 .

[136]  José G. Ramcrez.  Data Analysis: Statistical and Computational Methods for Scientists and Engineers , 2000, Technometrics.

[137]  Roy D. Williams,et al.  Estimating the Error of Numerical Solutions of Systems of Reaction-Diffusion Equations , 2000 .

[138]  J. Oden,et al.  A Posteriori Error Estimation in Finite Element Analysis , 2000 .

[139]  J. Oden,et al.  A Posteriori Error Estimation in Finite Element Analysis: Oden/A Posteriori , 2000 .

[140]  Jon C. Helton,et al.  Conceptual structure of the 1996 performance assessment for the Waste Isolation Pilot Plant , 2000, Reliab. Eng. Syst. Saf..

[141]  Tae M. Ahn,et al.  An approach to the assessment of high-level radioactive waste containment - II : radionuclide releases from an engineered barrier system , 2000 .

[142]  Jon C. Helton,et al.  Radionuclide transport in the vicinity of the repository and associated complementary cumulative distribution functions in the 1996 performance assessment for the Waste Isolation Pilot Plant , 2000, Reliab. Eng. Syst. Saf..

[143]  G. Duclos.  New York 1987 , 2000 .

[144]  Dominique Pelletier,et al.  VERIFICATION OF ERROR ESTIMATORS FOR THE EULER EQUATIONS , 2000 .

[145]  Timothy G. Trucano,et al.  Validation Methodology in Computational Fluid Dynamics , 2000 .

[146]  David P. Baumhefner,et al.  Ensemble simulations of explosive cyclogenesis at ranges of 2-5 days , 2000 .

[147]  Bernard P. Zeigler,et al.  Theory of Modeling and Simulation: Integrating Discrete Event and Continuous Complex Dynamic Systems , 2000 .

[148]  D. Venditti,et al.  Adjoint error estimation and grid adaptation for functional outputs: application to quasi-one-dimensional flow , 2000 .

[149]  Stephan Merz,et al.  Model Checking , 2000 .

[150]  Brian Skyrms,et al.  Bayes or Bust , 2000 .

[151]  Christopher J. Roy,et al.  Verification and Validation for Laminar Hypersonic Flowfields , 2000 .

[152]  A. Chlond,et al.  Large-Eddy Simulation Of A Nocturnal Stratocumulus-Topped Marine Atmospheric Boundary Layer: An Uncertainty Analysis , 2000 .

[153]  Patrick Knupp,et al.  Code Verification by the Method of Manufactured Solutions , 2000 .

[154]  T. Palmer.  Predicting uncertainty in forecasts of weather and climate , 2000 .

[155]  Hugh W. Coleman,et al.  Comprehensive Approach to Verification and Validation of CFD Simulations—Part 2: Application for Rans Simulation of a Cargo/Container Ship , 2001 .

[156]  Christopher J. Roy,et al.  Grid Convergence Error Analysis for Mixed-Order Numerical Schemes , 2001 .

[157]  R. Peyret,et al.  Computing singular solutions of the Navier–Stokes equations with the Chebyshev‐collocation method , 2001 .

[158]  Christopher J. Roy,et al.  Assessment of One- and Two-Equation Turbulence Models for Hypersonic Transitional Flows , 2001 .

[159]  J. Murzewski,et al.  Probability, Reliability and Statistical Methods in Engineering Design: A. Halder and S. Mahadevan, John Wiley & Sons, New York, 2000, xi+304 pp , 2001 .

[160]  Ivan Kramosil.  Probabilistic analysis of belief functions , 2001 .

[161]  Vicente J. Romero,et al.  Description of the Sandia Validation Metrics Project , 2001 .

[162]  James Glimm,et al.  Risk Management for Petroleum Reservoir Production: A Simulation-Based Study of Prediction , 2001 .

[163]  James Persse,et al.  Implementing the Capability Maturity Model , 2001 .

[164]  Timothy Hasselman.  Quantification of Uncertainty in Structural Dynamic Models , 2001 .

[165]  Quantitative Validation of Mathematical Models , 2001, Heat Transfer: Volume 5 — Computational, Aerospace and Environmental Heat Transfer.

[166]  I. Babuska,et al.  The finite element method and its reliability , 2001 .

[167]  A G Hutton,et al.  QUALITY AND TRUST IN INDUSTRIAL CFD - A EUROPEAN INITIATIVE , 2001 .

[168]  Robert G. Easterling,et al.  Measuring the Predictive Capability of Computational Models: Principles and Methods, Issues and Illustrations , 2001 .

[169]  Hugh W. Coleman,et al.  Comprehensive Approach to Verification and Validation of CFD Simulations—Part 1: Methodology and Procedures , 2001 .

[170]  Timothy G. Trucano,et al.  Statistical Validation of Engineering and Scientific Models with Application to CTH , 2001 .

[171]  Michael J. Hemsch,et al.  Statistical Analysis of CFD Solutions from the Drag Prediction Workshop , 2002 .

[172]  R W Logan,et al.  Verification& Validation (V&V) Guidelines and Quantitative Reliability at Confidence (QRC): Basis for an Investment Strategy , 2002 .

[173]  Brian E. Thompson,et al.  Assessment of Numerical Uncertainty Around Shocks and Corners on Blunt Trailing-Edge Supercritical Airfoils , 2002 .

[174]  J. Cadafalch,et al.  Verification of Finite Volume Computations on Steady-State Fluid Flow and Heat Transfer , 2002 .

[175]  Timothy G. Trucano,et al.  Verification and Validation in Computational Fluid Dynamics , 2002 .

[176]  Kathleen V. Diegert,et al.  Error and uncertainty in modeling and simulation , 2002, Reliab. Eng. Syst. Saf..

[177]  Patrick Knupp,et al.  Verification of Computer Codes in Computational Science and Engineering , 2002 .

[178]  Timothy G. Trucano,et al.  General Concepts for Experimental Validation of ASCI Code Applications , 2002 .

[179]  Roger U. Fujii.  Independent Verification and Validation , 2002 .

[180]  John C. Vassberg,et al.  Summary of Data from the First AIAA CFD Drag Prediction Workshop , 2002 .

[181]  Gerard O’Regan,et al.  The Capability Maturity Model , 2002 .

[182]  Boris Diskin,et al.  The NASA STI Program Office provides , 1997 .

[183]  Yoon-ha Lee,et al.  Uncertainty Quantification for Multiscale Simulations , 2002 .

[184]  Timothy G. Trucano,et al.  Statistical Validation of Engineering and Scientific Models: A Maximum Likelihood Based Metric , 2002 .

[185]  Ian H. Leslie,et al.  Statistical Validation of Engineering and Scientific Models: Validation Experiments to Application , 2003 .

[186]  Sankaran Mahadevan,et al.  Bayesian methodology for reliability model acceptance , 2003, Reliab. Eng. Syst. Saf..

[187]  Thomas L. Paez,et al.  Statistical Validation of Structural Dynamics Models , 2003 .

[188]  John Rushby,et al.  Formal Methods and the Certification of Critical Systems , 2004 .

[189]  W. Gilks.  Markov Chain Monte Carlo , 2005 .

[190]  J. Rosenthal,et al.  Markov Chain Monte Carlo , 2018 .

[191]  J. S. Turner,et al.  QUANTIFICATION OF UNCERTAINTY IN COMPUTATIONAL FLUID DYNAMICS , 2006 .

[192]  Severe Accident Risks : An Assessment for Five U . S . Nuclear Power Plants , 2006 .