Study of industrially applied methods for verification, validation and uncertainty quantification of simulator models

To better utilize the potential of system simulation models and simulators, industrially applicable methods for Verification, Validation and Uncertainty Quantification (VV&UQ) are crucial. This paper presents an exploratory case study of VV&UQ techniques applied on models integrated in aircraft system simulators at Saab Aeronautics and in driving simulators at the Swedish National Road and Transport Research Institute (VTI). Results show that a large number of Verification and Validation (V&V) techniques are applied, some of which are promising for further development and use in simulator credibility assessment. Regarding the application of UQ, a large gap between academia and this part of industry has been identified, and simplified methods are needed. The applicability of the NASA Credibility Assessment Scale (CAS) at the studied organizations is also evaluated and it can be concluded that the CAS is considered to be a usable tool for achieving a uniform level of V&V for all models included in a simulator, although its implementation at the studied organizations requires tailoring and coordination.

[1]  Henric Andersson,et al.  Variability and Customization of Simulator Products: A Product Line Approach in Model Based Systems Engineering , 2012 .

[2]  Osman Balci,et al.  Verification, Validation, and Testing , 2007 .

[3]  Magnus Carlsson,et al.  Methods for Early Model Validation : Applied on Simulation Models of Aircraft Vehicle Systems , 2013 .

[4]  Mark Kasunic,et al.  Designing an Effective Survey , 2005 .

[5]  Axel Lehmann,et al.  Quality Assurance of Models and simulation Applications , 2010, Int. J. Model. Simul. Sci. Comput..

[6]  Tuncer I. Ören,et al.  SIMULATION AND REALITY: THE BIG PICTURE , 2010 .

[7]  Sören Steinkellner,et al.  Aircraft Vehicle Systems Modeling and Simulation under Uncertainty , 2011 .

[8]  R. Yin.  Case Study Research: Design and Methods , 1984 .

[9]  Petter Krus,et al.  Modeling and simulation of Saab Gripen’s vehicle systems , 2009 .

[10]  Osman Balci,et al.  Verification, Validation And Accreditation Of Simulation Models , 1997, Winter Simulation Conference Proceedings,.

[11]  Magnus Carlsson,et al.  Enabling Uncertainty Quantification of Large Aircraft System Simulation Models , 2013 .

[12]  Christopher J. Roy,et al.  A comprehensive framework for verification, validation, and uncertainty quantification in scientific computing , 2011 .

[13]  Magnus Carlsson,et al.  Methodology for Development and Validation of Multipurpose Simulation Models , 2012 .

[14]  Per Runeson,et al.  Guidelines for conducting and reporting case study research in software engineering , 2009, Empirical Software Engineering.

[15]  Martin J. Steele.  Standard for Models and Simulations , 2016 .