Model Validation in Early Phase of Designing Complex Engineered Systems

In design process of a complex engineered system, studying the behavior of the system prior to manufacturing plays a key role to reduce cost of design and enhance the efficiency of the system during its lifecycle. To study the behavior of the system in the early design phase, it is required to model the characterization of the system and simulate the system’s behavior. The challenge is the fact that in early design stage, there is no or little information from the real system’s behavior, therefore there is not enough data to use to validate the model simulation and make sure that the model is representing the real system’s behavior appropriately. In this paper, we address this issue and propose methods to validate the model developed in the early design stage. First we propose a method based on FMEA and show how to quantify expert’s knowledge and validate the model simulation in the early design stage. Then, we propose a non-parametric technique to test if the observed behavior of one or more subsystems which currently exist, and the model simulation are the same. In addition, a local sensitivity analysis search tool is developed that helps the designers to focus on sensitive parts of the system in further design stages, particularly when mapping the conceptual model to a component model. We apply the proposed methods to validate the output of failure simulation developed in the early stage of designing a monopropellant propulsion system design.

[1]  Nathan Mantel,et al.  Chi-square tests with one degree of freedom , 1963 .

[2]  Liang Tang,et al.  Simulation-based Design and Validation of Automated Contingency Management for Propulsion Systems , 2007, 2007 IEEE Aerospace Conference.

[3]  Ramana V. Grandhi,et al.  Epistemic uncertainty quantification techniques including evidence theory for large-scale structures , 2004 .

[4]  J. Berger,et al.  The Intrinsic Bayes Factor for Model Selection and Prediction , 1996 .

[5]  M. Stone.  An Asymptotic Equivalence of Choice of Model by Cross‐Validation and Akaike's Criterion , 1977 .

[6]  Wasserman,et al.  Bayesian Model Selection and Model Averaging. , 2000, Journal of mathematical psychology.

[7]  David A. Schoenfeld,et al.  Chi-squared goodness-of-fit tests for the proportional hazards regression model , 1980 .

[8]  Averill M. Law,et al.  How to build valid and credible simulation models , 2008, 2008 Winter Simulation Conference.

[9]  Liang Tang,et al.  Prognostics in the Control Loop , 2007, AAAI Fall Symposium: Artificial Intelligence for Prognostics.

[10]  Kari Sentz,et al.  Combination of Evidence in Dempster-Shafer Theory , 2002 .

[11]  Bhaskar Saha,et al.  Using Fault Augmented Modelica Models for Diagnostics , 2014 .

[12]  Robert G. Sargent,et al.  An overview of verification and validation of simulation models , 1987, WSC '87.

[13]  Q. Vuong.  Likelihood Ratio Tests for Model Selection and Non-Nested Hypotheses , 1989 .

[14]  Malcolm J. Beynon,et al.  The Dempster-Shafer Theory , 2009, Encyclopedia of Artificial Intelligence.

[15]  W. Conover.  A Kolmogorov Goodness-of-Fit Test for Discontinuous Distributions , 1972 .

[16]  Lotfi A. Zadeh,et al.  Review of A Mathematical Theory of Evidence , 1984 .

[17]  Clifford M. Hurvich,et al.  Regression and time series model selection in small samples , 1989 .

[18]  D M Barends,et al.  Risk analysis by FMEA as an element of analytical validation. , 2009, Journal of pharmaceutical and biomedical analysis.

[19]  Ron Kohavi,et al.  A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection , 1995, IJCAI.

[20]  F. Massey.  The Kolmogorov-Smirnov Test for Goodness of Fit , 1951 .

[21]  Alexandre d'Aspremont,et al.  Model Selection Through Sparse Max Likelihood Estimation Model Selection Through Sparse Maximum Likelihood Estimation for Multivariate Gaussian or Binary Data , 2022 .

[22]  Darius Karkaria.  Independent Verification and Validation: a Life Cycle Engineering Process for Quality Software, by R. O. Lewis, Wiley, 1992 (Book Review) , 1993, Softw. Test. Verification Reliab..

[23]  Sylvain Arlot,et al.  A survey of cross-validation procedures for model selection , 2009, 0907.4728.

[24]  Irem Y. Tumer,et al.  Resilient System Design Using Cost-Risk Analysis With Functional Models , 2017, DAC 2017.

[25]  E. Triantaphyllou,et al.  A Sensitivity Analysis Approach for Some Deterministic Multi-Criteria Decision-Making Methods* , 1997 .

[26]  Sheng‐Hsien Teng,et al.  Failure mode and effects analysis: An integrated approach for product design and process control , 1996 .

[27]  P. Sanders,et al.  DoD Modeling and Simulation (M&S) Verification, Validation, and Accreditation (VV&A), , 1996 .

[28]  Wei Chen,et al.  A better understanding of model updating strategies in validating engineering models , 2008 .

[29]  David A. Cook.  How to Perform Credible Verification , Validation , and Accreditation for Modeling and Simulation , 2005 .

[30]  Xiaobo Zhou,et al.  Global Sensitivity Analysis , 2017, Encyclopedia of GIS.

[31]  R. Iman,et al.  Rank Transformations as a Bridge between Parametric and Nonparametric Statistics , 1981 .

[32]  P. L. Goddard.  Validating the safety of embedded real-time control systems using FMEA , 1993, Annual Reliability and Maintainability Symposium 1993 Proceedings.

[33]  H. Schneider.  Failure mode and effect analysis : FMEA from theory to execution , 1996 .

[34]  Padhraic Smyth,et al.  Model selection for probabilistic clustering using cross-validated likelihood , 2000, Stat. Comput..

[35]  Paola Annoni,et al.  Variance based sensitivity analysis of model output. Design and estimator for the total sensitivity index , 2010, Comput. Phys. Commun..

[36]  Jon C. Helton,et al.  Implementation and evaluation of nonparametric regression procedures for sensitivity analysis of computationally demanding models , 2009, Reliab. Eng. Syst. Saf..

[37]  Thierry Alex Mara,et al.  Variance-based sensitivity indices for models with dependent inputs , 2012, Reliab. Eng. Syst. Saf..

[38]  Caren Marzban,et al.  Variance-Based Sensitivity Analysis: An Illustration on the Lorenz'63 Model , 2013 .

[39]  A. Raftery.  Bayesian Model Selection in Social Research , 1995 .