Why Do Verification and Validation

In this discussion paper, we explore different ways to assess the value of verification and validation (V&V) of engineering models. We first present a literature review on the value of V&V and then use value chains and decision trees to show how value can be assessed from a decision maker's perspective. In this context, the value is what the decision maker is willing to pay for V&V analysis with the understanding that the V&V results are uncertain. As a result, the 2014 Sandia V&V Challenge Workshop is used to illustrate these ideas.

[1]  Paul Gray.  The economics of simulation , 1977, SIML.

[2]  Joseph H. Morrison,et al.  Towards a Credibility Assessment of Models and Simulations , 2008 .

[3]  Paul R. Muessig,et al.  Optimizing the selection of VV&A activities: a risk/benefit approach , 1997, WSC '97.

[4]  Joshua Mullins,et al.  Summary of the 2014 Sandia Verification and Validation Challenge Workshop , 2016 .

[5]  Leonard E. Schwer,et al.  An overview of the PTC 60/V&V 10: guide for verification and validation in computational solid mechanics , 2007, Engineering with Computers.

[6]  Arthur Rizzi,et al.  Toward Establishing Credibility in Computational Fluid Dynamics Simulations , 1998 .

[7]  John N. Tsitsiklis,et al.  Introduction to Probability , 2002 .

[8]  Gordon Grant,et al.  Building a Business Case for Modeling and Simulation , 2000 .

[9]  George Edgar Orient,et al.  The 2014 Sandia Verification and Validation Challenge: Problem Statement , 2016 .

[10]  James B. Dabney,et al.  Estimating direct return on investment of independent verification and validation , 2004, IASTED Conf. on Software Engineering and Applications.

[11]  Steve Gordon,et al.  Calculating Return on Investment for U.S. Department of Defense Modeling and Simulation , 2011 .

[12]  M. Porter.  Competitive Advantage: Creating and Sustaining Superior Performance , 1985 .

[13]  Philippe Artzner,et al.  Coherent Measures of Risk , 1999 .

[14]  Robert T. Clemen,et al.  Making Hard Decisions: An Introduction to Decision Analysis , 1997 .

[15]  Roger W. Logan,et al.  Risk Reduction as the Product of Model Assessed Reliability, Confidence, and Consequence , 2005 .

[16]  Christopher J. Roy,et al.  Probability Bounds Analysis Applied to the Sandia Verification and Validation Challenge Problem , 2016 .

[17]  Thomas L. Paez,et al.  Economic Analysis of Model Validation for a Challenge Problem , 2016 .

[18]  Ilan Kelman,et al.  Does mitigation save? Reviewing cost-benefit analyses of disaster risk reduction , 2014 .

[19]  Robin L. Dillon,et al.  The Respective Roles of Risk and Decision Analyses in Decision Support , 2006, Decis. Anal..

[20]  J. Berger.  Statistical Decision Theory and Bayesian Analysis , 1988 .

[21]  James N. Elele,et al.  Risk-based verification, validation, and accreditation process , 2010, Defense + Commercial Sensing.

[22]  S. Mahadevan,et al.  Bayesian Uncertainty Integration for Model Calibration, Validation, and Prediction , 2016 .

[23]  Osman Balci,et al.  A methodology for cost-risk analysis in the statistical validation of simulation models , 1981, CACM.

[24]  François M. Hemez,et al.  Defining predictive maturity for validated numerical simulations , 2010 .

[25]  Phillip J. Lederer,et al.  Economics of total quality management , 1995 .

[26]  Vicente J. Romero,et al.  Introduction: The 2014 Sandia Verification and Validation Challenge Workshop , 2016 .

[27]  Patricia Diane Hough,et al.  Sandia Verification and Validation Challenge Problem: A PCMM-Based Approach to Assessing Prediction Credibility , 2016 .

[28]  Ren-Jye Yang,et al.  Reliability Analysis With Model Uncertainty Coupling With Parameter and Experiment Uncertainties: A Case Studyof 2014 Verification and Validation Challenge Problem , 2016 .

[29]  Tarek K. Abdel-Hamid,et al.  The Economics of Software Quality Assurance: A Simulation-Based Case Study , 1988, MIS Q..

[30]  D. Pace.  Modeling and simulation verification and validation challenges , 2004 .

[31]  Wei Li,et al.  Integrating Bayesian Calibration, Bias Correction, and Machine Learning for the 2014 Sandia Verification and Validation Challenge Problem , 2016 .

[32]  Simon J. E. Taylor,et al.  Economics of modeling and simulation: Reflections and implications for healthcare , 2010, Proceedings of the 2010 Winter Simulation Conference.