Computer experiment designs for accurate prediction

Computer experiments using deterministic simulators are sometimes used to replace or supplement physical system experiments. This paper compares designs for an initial computer simulator experiment based on empirical prediction accuracy; it recommends designs for producing accurate predictions. The basis for the majority of the designs compared is the integrated mean squared prediction error (IMSPE) that is computed assuming a Gaussian process model with a Gaussian correlation function. Designs that minimize the IMSPE with respect to a fixed set of correlation parameters as well as designs that minimize a weighted IMSPE over the correlation parameters are studied. These IMSPE-based designs are compared with three widely-used space-filling designs. The designs are used to predict test surfaces representing a range of stationary and non-stationary functions. For the test conditions examined in this paper, the designs constructed under IMSPE-based criteria are shown to outperform space-filling Latin hypercube designs and maximum projection designs when predicting smooth functions of stationary appearance, while space-filling and maximum projection designs are superior for test functions that exhibit strong non-stationarity.

[1]  T. Laursen,et al.  Finite Element Modeling Predictions of Region-specific Cell-matrix Mechanics in the Meniscus , 2006, Biomechanics and modeling in mechanobiology.

[2]  Thomas J. Santner,et al.  Robust design for acetabular cup stability accounting for patient and surgical variability. , 2008, Journal of biomechanical engineering.

[3]  D. Montgomery,et al.  Comparing Computer Experiments for the Gaussian Process Model Using Integrated Prediction Variance , 2013 .

[4]  Geof H. Givens,et al.  Computational Statistics: Givens/Computational Statistics , 2012 .

[5]  Srikant Nekkanty,et al.  Characterization of damage and optimization of thin film coatings on ductile substrates , 2009 .

[6]  Luc Pronzato,et al.  Design of computer experiments: space filling and beyond , 2011, Statistics and Computing.

[7]  Michael W. Trosset.  The Krigifier: A Procedure for Generating Pseudorandom Nonlinear Objective Functions for Computational Experimentation , 1999 .

[8]  Lynn T. Truss,et al.  Combining a Statistical Design of Experiments with Formability Simulations to Predict the Formability of Pockets in Sheet Metal Parts , 2001 .

[9]  T. J. Mitchell,et al.  Exploratory designs for computational experiments , 1995 .

[10]  B. Efron.  Frequentist accuracy of Bayesian estimates , 2015, Journal of the Royal Statistical Society. Series B, Statistical methodology.

[11]  Thomas J. Santner,et al.  Computer Experiment Designs via Particle Swarm Optimization , 2014 .

[12]  Jerome Sacks,et al.  Designs for Computer Experiments , 1989 .

[13]  Thomas J. Santner,et al.  The Design and Analysis of Computer Experiments , 2003, Springer Series in Statistics.

[14]  Hugh A. Chipman,et al.  GPfit: An R Package for Fitting a Gaussian Process Model to Deterministic Simulator Outputs , 2013, 1305.0759.

[15]  Hongqiang Guo,et al.  Using a statistically calibrated biphasic finite element model of the human knee joint to identify robust designs for a meniscal substitute. , 2014, Journal of biomechanical engineering.

[16]  M. D. McKay,et al.  A comparison of three methods for selecting values of input variables in the analysis of output from a computer code , 2000 .

[17]  Andy J. Keane,et al.  Engineering Design via Surrogate Modelling - A Practical Guide , 2008 .

[18]  David M. Steinberg,et al.  Comparison of designs for computer experiments , 2006 .

[19]  Dave Higdon,et al.  Combining Field Data and Computer Simulations for Calibration and Prediction , 2005, SIAM J. Sci. Comput..

[20]  V. Roshan Joseph,et al.  Composite Gaussian process models for emulating expensive functions , 2012, 1301.2503.

[21]  Aaron L. Fogelson,et al.  Computational Modeling of Blood Clotting: Coagulation and Three-dimensional Platelet Aggregation , 2003 .

[22]  Fangfang Sun,et al.  A comparison of two metamodel-based methodologies for multiple criteria simulation optimization using an injection molding case study , 2013 .

[23]  W. Welch.  ACED: Algorithms for the Construction of Experimental Designs , 1985 .

[24]  M. Liefvendahl,et al.  A study on algorithms for optimization of Latin hypercubes , 2006 .

[25]  T. J. Mitchell,et al.  Bayesian Prediction of Deterministic Functions, with Applications to the Design and Analysis of Computer Experiments , 1991 .

[26]  W. Cheney,et al.  Numerical Analysis: Mathematics of Scientific Computing , 1991 .

[27]  Johann Sienz,et al.  Formulation of the Optimal Latin Hypercube Design of Experiments Using a Permutation Genetic Algorithm , 2004 .

[28]  Harald Niederreiter,et al.  Random number generation and Quasi-Monte Carlo methods , 1992, CBMS-NSF regional conference series in applied mathematics.

[29]  A. Tamhane.  Design and Analysis of Experiments for Statistical Selection, Screening, and Multiple Comparisons , 1995 .

[30]  R. Caflisch,et al.  Quasi-Monte Carlo integration , 1995 .

[31]  M. E. Johnson,et al.  Minimax and maximin distance designs , 1990 .

[32]  James Kennedy,et al.  Particle swarm optimization , 1995, Proceedings of ICNN'95 - International Conference on Neural Networks.

[33]  V. R. Joseph,et al.  Maximum projection designs for computer experiments , 2015 .

[34]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[35]  Douglas C. Montgomery,et al.  An empirical study of the prediction performance of space-filling designs , 2011 .

[36]  Xin-She Yang,et al.  Engineering Optimization: An Introduction with Metaheuristic Applications , 2010 .

[37]  Thomas J. Santner,et al.  Noncollapsing Space-Filling Designs for Bounded Nonrectangular Regions , 2012, Technometrics.

[38]  S. Bates,et al.  Formulation of the Audze--Eglais uniform Latin hypercube design of experiments , 2003 .

[39]  Wei Chen,et al.  A non‐stationary covariance‐based Kriging method for metamodelling in engineering design , 2007 .