Error and Uncertainty Analysis of Inexact and Imprecise Computer Models

Computer simulations are routinely executed to predict the behavior of complex systems in many fields of engineering and science. These computer-aided predictions involve the theoretical foundation, numerical modeling, and supporting experimental data, all of which come with their associated errors. A natural question then arises concerning the validity of computer model predictions, especially in cases where these models are executed in support of high-consequence decision making. This article lays out a methodology for quantifying the degrading effects of incompleteness and inaccuracy of the theoretical foundation, numerical modeling, and experimental data on the computer model predictions. Through the method discussed in this paper, the validity of model predictions can be judged and communicated between involved parties in a quantitative and objective manner. DOI: 10.1061/(ASCE)CP.1943-5487.0000233. © 2013 American Society of Civil Engineers. CE Database subject headings: Optimization; Uncertainty principles; Parameters; Calibration; Validation; Errors; Computer models; Computer aided simulation. Author keywords: Optimization; Uncertainty propagation; Discrepancy bias; Parameter calibration; Bias correction; Model validation; Model form error; Test-analysis correlation.

[1]  D Sornette,et al.  Algorithm for model validation: Theory and applications , 2007, Proceedings of the National Academy of Sciences.

[2]  A. O'Hagan,et al.  Bayesian calibration of computer models , 2001 .

[3]  Bo Wang,et al.  Bayesian variable selection for Gaussian process regression: Application to chemometric calibration of spectrometers , 2010, Neurocomputing.

[4]  Raghavan Srinivasan,et al.  Evaluation of global optimization algorithms for parameter calibration of a computationally intensive hydrologic model , 2009 .

[5]  François M. Hemez,et al.  A forecasting metric for predictive modeling , 2011 .

[6]  Zong Woo Geem,et al.  A New Heuristic Optimization Algorithm: Harmony Search , 2001, Simul..

[7]  Chris W. Johnson,et al.  Tools for Local Critical Infrastructure Protection : Computational Support for Identifying Safety and Security Interdependencies between Local Critical Infrastructures , 2008 .

[8]  W C Felch,et al.  Professional competence. , 1973, California medicine.

[9]  Lu Xu,et al.  Automatic configuration of optimized sample-weighted least-squares support vector machine by particle swarm optimization for multivariate spectral analysis , 2010 .

[10]  Emile H. L. Aarts,et al.  Towards implementing the algorithm , 1987 .

[11]  J. R. Kamm,et al.  A Brief Overview of the State-of-the-Practice and Current Challenges of Solution Verification , 2008 .

[12]  William L. Oberkampf,et al.  Guide for the verification and validation of computational fluid dynamics simulations , 1998 .

[13]  Manolis Papadrakakis,et al.  A Hybrid Particle Swarm—Gradient Algorithm for Global Structural Optimization , 2010, Comput. Aided Civ. Infrastructure Eng..

[14]  Dave Higdon,et al.  Combining Field Data and Computer Simulations for Calibration and Prediction , 2005, SIAM J. Sci. Comput..

[15]  David Draper,et al.  Assessment and Propagation of Model Uncertainty , 2011 .

[16]  François M. Hemez,et al.  Defining predictive maturity for validated numerical simulations , 2010 .

[17]  J R Saunders,et al.  A particle swarm optimizer with passive congregation. , 2004, Bio Systems.

[18]  Baher Abdulhai,et al.  Genetic algorithm-based combinatorial parametric optimization for the calibration of microscopic traffic simulation models , 2001, ITSC 2001. 2001 IEEE Intelligent Transportation Systems. Proceedings (Cat. No.01TH8585).

[19]  Nikolaos Scarmeas,et al.  The good, bad, and ugly? , 2012, Neurology.

[20]  Ioannis G. Tsoulos,et al.  Enhancing PSO methods for global optimization , 2010, Appl. Math. Comput..

[21]  Christopher J. Roy,et al.  A Complete Framework for Verification, Validation, and Uncertainty Quantification in Scientific Computing (Invited) , 2010 .

[22]  ChangKyoo Yoo,et al.  A systematic model calibration methodology based on multiple errors minimization method for the optimal parameter estimation of ASM1 , 2012, Korean Journal of Chemical Engineering.

[23]  François M. Hemez,et al.  The Good , The Bad , and The Ugly of Predictive Science , 2005 .

[24]  Russell C. Eberhart,et al.  A new optimizer using particle swarm theory , 1995, MHS'95. Proceedings of the Sixth International Symposium on Micro Machine and Human Science.

[25]  J. J. Rowland,et al.  Professional competence in safety-related software engineering , 1995, Softw. Eng. J..

[26]  Michael J. Markow.  Computer Simulation to Address Issues of Public Policy , 1984 .

[27]  David A. Swayne,et al.  A Novel Model Calibration Technique Through Application of Machine Learning Association Rules , 2010 .

[28]  Katherine Campbell,et al.  Statistical calibration of computer simulations , 2006, Reliab. Eng. Syst. Saf..

[29]  Terje Haukaas,et al.  Model Uncertainty in Finite-Element Analysis: Bayesian Finite Elements , 2011 .

[30]  Robert V. Hogg,et al.  Introduction to Mathematical Statistics. , 1966 .

[31]  Yuhui Shi,et al.  Particle swarm optimization: developments, applications and resources , 2001, Proceedings of the 2001 Congress on Evolutionary Computation (IEEE Cat. No.01TH8546).

[32]  C. Lucas,et al.  Automatic Calibration of Lumped Conceptual Rainfall-Runoff Model Using Particle Swarm Optimization , 2008 .

[33]  Emile H. L. Aarts,et al.  Simulated Annealing: Theory and Applications , 1987, Mathematics and Its Applications.

[34]  Fang Zhao,et al.  3D Traffic Simulation for Intermodal Safety and Security , 2006 .

[35]  K. Lee,et al.  A new structural optimization method based on the harmony search algorithm , 2004 .

[36]  Charles E. Augarde,et al.  Simulation-based calibration of geotechnical parameters using parallel hybrid moving boundary particle swarm optimization , 2009 .

[37]  Luca Maria Gambardella,et al.  Ant Algorithms for Discrete Optimization , 1999, Artificial Life.

[38]  G. Reinsel,et al.  Introduction to Mathematical Statistics (4th ed.). , 1980 .

[39]  Michael N. Vrahatis,et al.  Parameter selection and adaptation in Unified Particle Swarm Optimization , 2007, Math. Comput. Model..

[40]  Michael Andrew Christie,et al.  Error analysis and simulations of complex phenomena , 2005 .

[41]  Brian Williams,et al.  A Bayesian calibration approach to the thermal problem , 2008 .

[42]  C. D. Gelatt,et al.  Optimization by Simulated Annealing , 1983, Science.

[43]  Jeremy E. Oakley,et al.  Bayesian Analysis of Computer Model Outputs , 2002 .

[44]  James O. Berger,et al.  A Framework for Validation of Computer Models , 2007, Technometrics.

[45]  Laura Painton Swiler,et al.  Calibration, validation, and sensitivity analysis: What's what , 2006, Reliab. Eng. Syst. Saf..

[46]  Byungchang Jung.  A hierarchical framework for statistical model validation of engineered systems , 2011 .

[47]  Xiaodong Wang,et al.  Particle Swarm Optimization for Calibrating Stream Water Quality Model , 2008, 2008 Second International Symposium on Intelligent Information Technology Application.