Assessing risk levels of verification, validation, and accreditation of models and simulations

This paper presents a new methodology for assessing the risks associated with the level of Verification, Validation and Accreditation (VV&A) of a given Model and/or Simulation when used in support of major decisions. As stated by DoD Instruction 5000.61 , "It is DoD policy that: Models and Simulations (M&S) used to support major DoD decision-making organizations and processes shall be accredited for that specific purpose by the DoD Component M&S Application Sponsor." This Instruction applies to "All models and simulations developed, used, or managed by the DoD Components after the effective date of this Instruction." The requirements cited above have set the need for VV&A of M&S at the forefront of concerns for DoD and DoD-Components acquisition personnel. When an acquisition program involves a large number of models, cost associated with VV&A can become enormous. There is a need therefore to have a systematic approach for assessing and prioritizing the risks associated with the level to which individual models have been verified, validated, and accredited. To provide decision makers with a judicious way for determining the risks associated with using a given M&S, and the extent to which VV&A work will be needed to meet these requirements, we have developed a methodology for assessing the risks associated with the level of VV&A of a given M&S when used to support decision-making. This approach parallels the formal DoD Risk Assessment procedure, but with application to the use of M&S, as it relates to VV&A. Risks associated with the levels of VV&A are evaluated and assessed based on the following criteria: - Likelihood of the M&S being inaccurate and/or inappropriate for the application. - Consequences of the M&S being inaccurate and/or inappropriate for the application. - Importance of the (acquisition) decision supported by the M&S. - Level of Reliance of the (acquisition) decision on the M&S. The assessment results are used to classify individual models into three risk categories (Red for High Risk, Yellow for Medium Risk, and Green for Low Risk). All models classified as High Risk, are subjected to further analysis and recommendations made as to further work needed to reduce the risk to an acceptable level through formal VV&A. Models assessed as Medium Risk are also further evaluated and recommendations made with regards to the need for further risk mitigation. Models rated Low Risk are either not of primary importance in supporting the (acquisition) decisions being made, or they may have been assessed to have been satisfactorily verified, validated, and possibly accredited to an acceptable level.