Verification Methods for Surrogate Models

A surrogate model built based on a limited number of sample points will inevitably have large prediction uncertainty. Applying such imprecise surrogate models in design and optimization may lead to misleading predictions or optimal solutions located in unfeasible regions (Picheny in Improving accuracy and compensating for uncertainty in surrogate modeling. University of Florida, Gainesville, 2009). Therefore, verifying the accuracy of a surrogate model before using it can ensure the reliability of the design.

[1]  T. Simpson,et al.  Comparative studies of metamodelling techniques under multiple modelling criteria , 2001 .

[2]  Yuhong Yang.  CONSISTENCY OF CROSS VALIDATION FOR COMPARING REGRESSION PROCEDURES , 2007, 0803.2963.

[3]  Ron Kohavi,et al.  A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection , 1995, IJCAI.

[4]  Rupert G. Miller.  The jackknife-a review , 1974 .

[5]  Sylvain Arlot,et al.  A survey of cross-validation procedures for model selection , 2009, 0907.4728.

[6]  Neil Salkind,et al.  Encyclopedia of research design , 2010 .

[7]  M. H. Quenouille.  Approximate tests of correlation in time-series 3 , 1949, Mathematical Proceedings of the Cambridge Philosophical Society.

[8]  Nielen Stander,et al.  Comparing three error criteria for selecting radial basis function network topology , 2009 .

[9]  Agostino Di Ciaccio,et al.  Computational Statistics and Data Analysis Measuring the Prediction Error. a Comparison of Cross-validation, Bootstrap and Covariance Penalty Methods , 2022 .

[10]  Rob J Hyndman,et al.  Another look at measures of forecast accuracy , 2006 .

[11]  T. Chai,et al.  Root mean square error (RMSE) or mean absolute error (MAE)? – Arguments against avoiding RMSE in the literature , 2014 .

[12]  Wentao Mao,et al.  A fast and robust model selection algorithm for multi-input multi-output support vector machine , 2014, Neurocomputing.

[13]  Haitao Liu,et al.  An adaptive sampling approach for Kriging metamodeling by maximizing expected prediction error , 2017, Comput. Chem. Eng..

[14]  Jin Li,et al.  A review of comparative studies of spatial interpolation methods in environmental sciences: Performance and impact factors , 2011, Ecol. Informatics.

[15]  Hui Zhou,et al.  An active learning variable-fidelity metamodelling approach based on ensemble of metamodels and objective-oriented sequential sampling , 2016 .

[16]  José Antonio Lozano,et al.  Sensitivity Analysis of k-Fold Cross Validation in Prediction Error Estimation , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[17]  J. Shao.  Bootstrap Model Selection , 1996 .

[18]  Philip Hans Franses,et al.  A note on the Mean Absolute Scaled Error , 2015 .

[19]  S. Larson.  The shrinkage of the coefficient of multiple correlation. , 1931 .

[20]  Biswarup Bhattacharyya,et al.  A Critical Appraisal of Design of Experiments for Uncertainty Quantification , 2018 .

[21]  Tadayoshi Fushiki,et al.  Estimation of prediction error by using K-fold cross-validation , 2011, Stat. Comput..

[22]  Hirokazu Yanagihara,et al.  Bias correction of cross-validation criterion based on Kullback-Leibler information under a general condition , 2006 .

[23]  R. Tibshirani,et al.  Improvements on Cross-Validation: The 632+ Bootstrap Method , 1997 .

[24]  E. Acar.  Various approaches for constructing an ensemble of metamodels using local measures , 2010 .

[25]  Raphael T. Haftka,et al.  Surrogate-based Analysis and Optimization , 2005 .

[26]  Qun Liu,et al.  Comparison of Akaike information criterion (AIC) and Bayesian information criterion (BIC) in selection of stock–recruitment relationships , 2006 .

[27]  Yan Wang,et al.  A sequential multi-fidelity metamodeling approach for data regression , 2017, Knowl. Based Syst..

[28]  G. Breukelen.  Analysis of covariance (ANCOVA) , 2010 .

[29]  P. Burman.  A comparative study of ordinary cross-validation, v-fold cross-validation and the repeated learning-testing methods , 1989 .

[30]  Salvador A. Pintos,et al.  Toward an optimal ensemble of kernel-based approximations with engineering applications , 2006 .

[31]  J. Barrett.  The Coefficient of Determination—Some Limitations , 1974 .

[32]  R. Haftka,et al.  Multiple surrogates: how cross-validation errors can help us to obtain the best predictor , 2009 .

[33]  Aki Vehtari,et al.  Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC , 2015, Statistics and Computing.

[34]  Selen Cremaschi,et al.  Adaptive sequential sampling for surrogate model generation with artificial neural networks , 2014, Comput. Chem. Eng..

[35]  C. Willmott,et al.  Advantages of the mean absolute error (MAE) over the root mean square error (RMSE) in assessing average model performance , 2005 .

[36]  Cristina H. Amon,et al.  Error Metrics and the Sequential Refinement of Kriging Metamodels , 2015 .

[37]  Pengcheng Ye,et al.  Ensemble of surrogate based global optimization methods using hierarchical design space reduction , 2018 .

[38]  Yvan Saeys,et al.  An alternative approach to avoid overfitting for surrogate models , 2011, Proceedings of the 2011 Winter Simulation Conference (WSC).

[39]  Komahan Boopathy,et al.  Unified Framework for Training Point Selection and Error Estimation for Surrogate Models , 2015 .

[40]  M. Stone.  Cross‐Validatory Choice and Assessment of Statistical Predictions , 1976 .

[41]  T. Simpson,et al.  Computationally Inexpensive Metamodel Assessment Strategies , 2002 .

[42]  M. Victoria-Feser,et al.  A Robust Coefficient of Determination for Regression , 2010 .

[43]  M. Rais-Rohani,et al.  Ensemble of metamodels with optimized weight factors , 2008 .

[44]  Yahui Zhang,et al.  Comparative studies of error metrics in variable fidelity model uncertainty quantification , 2018, Journal of Engineering Design.

[45]  R. Haftka,et al.  Ensemble of surrogates , 2007 .

[46]  Eric-Jan Wagenmakers,et al.  Limitations of Bayesian Leave-One-Out Cross-Validation for Model Selection , 2018, Computational brain & behavior.

[47]  B. Efron.  Bootstrap Methods: Another Look at the Jackknife , 1979 .

[48]  J. Shao,et al.  The jackknife and bootstrap , 1996 .

[49]  Qiang Du,et al.  Centroidal Voronoi Tessellations: Applications and Algorithms , 1999, SIAM Rev..

[50]  A. Messac,et al.  Predictive quantification of surrogate model fidelity based on modal variations with sample density , 2015 .

[51]  Erdem Acar,et al.  Effect of error metrics on optimum weight factor selection for ensemble of metamodels , 2015, Expert Syst. Appl..

[52]  N. Nagelkerke,et al.  A note on a general definition of the coefficient of determination , 1991 .

[53]  Dong Zhao,et al.  A comparative study of metamodeling methods considering sample quality merits , 2010 .

[54]  J. Shao.  Linear Model Selection by Cross-validation , 1993 .

[55]  Raphael T. Hafkta,et al.  Comparing error estimation measures for polynomial and kriging approximation of noise-free functions , 2009 .

[56]  L. Breiman.  Heuristics of instability and stabilization in model selection , 1996 .

[57]  B. Efron.  Estimating the Error Rate of a Prediction Rule: Improvement on Cross-Validation , 1983 .

[58]  V. Picheny.  Improving accuracy and compensating for uncertainty in surrogate modeling , 2009 .