Computational methods for model reliability assessment

This paper investigates various statistical approaches for the validation of computational models when both model prediction and experimental observation have uncertainties, and proposes two new methods for this purpose. The first method utilizes hypothesis testing to accept or reject a model at a desired significance level. Interval-based hypothesis testing is found to be more practically useful for model validation than the commonly used point null hypothesis testing. Both classical and Bayesian approaches are investigated. The second and more direct method formulates model validation as a limit state-based reliability estimation problem. Both simulation-based and analytical methods are presented to compute the model reliability for single or multiple comparisons of the model output and observed data. The proposed methods are illustrated and compared using numerical examples.

[1]  Matthew F. Barone,et al.  Measures of agreement between computation and experiment: Validation metrics , 2004, J. Comput. Phys..

[2]  J. Berger,et al.  [Testing Precise Hypotheses]: Rejoinder , 1987 .

[3]  Keng C. Yap,et al.  Assessment of model accuracy relative to stochastic system behavior. , 2003 .

[4]  Jack P. C. Kleijnen,et al.  EUROPEAN JOURNAL OF OPERATIONAL , 1992 .

[5]  C. Spearman The proof and measurement of association between two things. , 2015, International journal of epidemiology.

[6]  Timothy G. Trucano,et al.  Verification and Validation in Computational Fluid Dynamics , 2002 .

[7]  Jacob Cohen The earth is round (p < .05) , 1994 .

[8]  Ian H. Leslie,et al.  Statistical Validation of Engineering and Scientific Models: Validation Experiments to Application , 2003 .

[9]  Ian H. Leslie,et al.  Case study for model validation : assessing a model for thermal decomposition of polyurethane foam. , 2004 .

[10]  Robert Tibshirani,et al.  An Introduction to the Bootstrap , 1994 .

[11]  Sankaran Mahadevan,et al.  Bayesian methodology for reliability model acceptance , 2003, Reliab. Eng. Syst. Saf..

[12]  D. Giles,et al.  A SADDLEPOINT APPROXIMATION TO THE DISTRIBUTION FUNCTION OF THE ANDERSON-DARLING TEST STATISTIC , 2001 .

[13]  Hugh W. Coleman,et al.  Uncertainties and CFD Code Validation , 1997 .

[14]  Sankaran Mahadevan,et al.  Bayesian risk-based decision method for model validation under uncertainty , 2007, Reliab. Eng. Syst. Saf..

[15]  Thaweepat Buranathiti,et al.  Model validation via uncertainty propagation and data transformations , 2004 .

[16]  M. Kendall Statistical Methods for Research Workers , 1937, Nature.

[17]  Sankaran Mahadevan,et al.  Model Predictive Capability Assessment Under Uncertainty , 2005 .

[18]  M. Schervish Theory of Statistics , 1995 .

[19]  Thomas A. Cruse,et al.  Reliability-Based Mechanical Design , 1997 .

[20]  Sankaran Mahadevan,et al.  Validation and error estimation of computational models , 2006, Reliab. Eng. Syst. Saf..

[21]  Robert Rosenthal,et al.  Interpretation of significance levels and effect sizes by psychological researchers. , 1986 .

[22]  J. Berger,et al.  Testing Precise Hypotheses , 1987 .

[23]  Osman Balci,et al.  Verification, Validation And Accreditation Of Simulation Models , 1997, Winter Simulation Conference Proceedings,.

[24]  Thomas L. Paez,et al.  On the Validation of a Mathematical Model of a Foam Encapsulated System , 2006 .

[25]  J. Berger,et al.  Testing a Point Null Hypothesis: The Irreconcilability of P Values and Evidence , 1987 .

[26]  H. Jeffreys,et al.  The Theory of Probability , 1896 .

[27]  A. W. Kemp,et al.  Kendall's Advanced Theory of Statistics. , 1994 .

[28]  R. Kirk Practical Significance: A Concept Whose Time Has Come , 1996 .

[29]  Dave Higdon,et al.  Combining Field Data and Computer Simulations for Calibration and Prediction , 2005, SIAM J. Sci. Comput..

[30]  Roger W. Logan,et al.  Comparing 10 Methods for Solution Verification, and Linking to Model Validation , 2005 .