Computer experiments: a review

In this paper we provide a broad introduction to the topic of computer experiments. We begin by briefly presenting a number of applications with different types of output or different goals. We then review modelling strategies, including the popular Gaussian process approach, as well as variations and modifications. Other strategies that are reviewed are based on polynomial regression, non-parametric regression and smoothing spline ANOVA. The issue of multi-level models, which combine simulators of different resolution in the same experiment, is also addressed. Special attention is given to modelling techniques that are suitable for functional data. To conclude the modelling section, we discuss calibration, validation and verification. We then review design strategies including Latin hypercube designs and space-filling designs and their adaptation to computer experiments. We comment on a number of special issues, such as designs for multi-level simulators, nested factors and determination of experiment size.

[1]  J. Halton.  On the efficiency of certain quasi-random sequences of points in evaluating multi-dimensional integrals , 1960 .

[2]  G. Matheron.  Principles of geostatistics , 1963 .

[3]  I. Sobol.  On the distribution of points in a cube and the approximate evaluation of integrals , 1967 .

[4]  Gunnar Abrahamson,et al.  Terminology for model credibility , 1980 .

[5]  J. Friedman,et al.  Projection Pursuit Regression , 1981 .

[6]  Henry P. Wynn,et al.  Maximum entropy sampling , 1987 .

[7]  A. O'Hagan.  [Design and Analysis of Computer Experiments]: Comment , 1989 .

[8]  M. Stein.  [Design and Analysis of Computer Experiments]: Comment , 1989 .

[9]  M. E. Johnson,et al.  Minimax and maximin distance designs , 1990 .

[10]  J. Friedman.  Multivariate adaptive regression splines , 1990 .

[11]  T. J. Mitchell,et al.  Bayesian Prediction of Deterministic Functions, with Applications to the Design and Analysis of Computer Experiments , 1991 .

[12]  Harald Niederreiter,et al.  Random number generation and Quasi-Monte Carlo methods , 1992, CBMS-NSF regional conference series in applied mathematics.

[13]  Henry P. Wynn,et al.  Screening, predicting, and computer experiments , 1992 .

[14]  Noel A Cressie,et al.  Statistics for Spatial Data. , 1992 .

[15]  Mike Rees,et al.  5. Statistics for Spatial Data , 1993 .

[16]  Boxin Tang.  Orthogonal Array-Based Latin Hypercubes , 1993 .

[17]  Eva Riccomagno,et al.  Experimental Design and Observation for Large Systems , 1996, Journal of the Royal Statistical Society: Series B (Methodological).

[18]  J. Faraway.  Regression analysis for a functional response , 1997 .

[19]  P. Roache.  Verification of Codes and Calculations , 1998 .

[20]  Jack P. C. Kleijnen,et al.  Improved Design of Queueing Simulation Experiments with Highly Heteroscedastic Responses , 1999, Oper. Res..

[21]  M. D. McKay,et al.  A comparison of three methods for selecting values of input variables in the analysis of output from a computer code , 2000 .

[22]  Yong Zhang,et al.  Uniform Design: Theory and Application , 2000, Technometrics.

[23]  A. O'Hagan,et al.  Predicting the output from a complex computer code when fast approximations are available , 2000 .

[24]  Michael Goldstein,et al.  Bayesian Forecasting for Complex Systems Using Computer Simulators , 2001 .

[25]  van der,et al.  Proceedings of the 2012 winter simulation conference , 2001, WSC 2008.

[26]  B. Silverman,et al.  Functional Data Analysis , 1997 .

[27]  Peter C. Young,et al.  The Identification and Estimation of Nonlinear Stochastic Systems , 2001 .

[28]  Thomas J. Santner,et al.  The Design and Analysis of Computer Experiments , 2003, Springer Series in Statistics.

[29]  Theodore T. Allen,et al.  Constructing Meta-Models for Computer Experiments , 2003 .

[30]  David M. Steinberg,et al.  Data Analytic Tools for Understanding Random Field Regression Models , 2004, Technometrics.

[31]  James O. Berger,et al.  Statistical Inverse Analysis for a Network Microsimulator , 2005, Technometrics.

[32]  Runze Li,et al.  Design and Modeling for Computer Experiments , 2005 .

[33]  Russell R. Barton,et al.  A review on design, modeling and applications of computer experiments , 2006 .

[34]  David M. Steinberg,et al.  Comparison of designs for computer experiments , 2006 .

[35]  Kenny Q. Ye,et al.  Variable Selection for Gaussian Process Models in Computer Experiments , 2006, Technometrics.

[36]  Thomas Hochkirchen.  Design and Modeling for Computer Experiments by K.-T. Fang, R. Li and A. Sudjianto , 2006 .

[37]  Dennis K. J. Lin,et al.  A construction method for orthogonal Latin hypercube designs , 2006 .

[38]  James O. Ramsay,et al.  Functional Data Analysis , 2005 .

[39]  J. Rougier,et al.  Bayes Linear Calibrated Prediction for Complex Systems , 2006 .

[40]  Dorin Drignei.  Empirical Bayesian Analysis for High-Dimensional Computer Output , 2006, Technometrics.

[41]  Robert B. Gramacy,et al.  Ja n 20 08 Bayesian Treed Gaussian Process Models with an Application to Computer Modeling , 2009 .

[42]  M. J. Bayarri,et al.  Computer model validation with functional output , 2007, 0711.3271.

[43]  David M. Steinberg,et al.  Modeling Data from Computer Experiments: An Empirical Comparison of Kriging with MARS and Projection Pursuit Regression , 2007 .

[44]  James O. Berger,et al.  A Framework for Validation of Computer Models , 2007, Technometrics.

[45]  D. Higdon,et al.  Computer Model Calibration Using High-Dimensional Output , 2008 .

[46]  Peter Z. G. Qian,et al.  Gaussian Process Models for Computer Experiments With Qualitative and Quantitative Factors , 2008, Technometrics.

[47]  Peter Z. G. Qian,et al.  Bayesian Hierarchical Modeling for Integrating Low-Accuracy and High-Accuracy Experiments , 2008, Technometrics.

[48]  J. Rougier.  Efficient Emulators for Multivariate Deterministic Functions , 2008 .

[49]  Agus Sudjianto,et al.  Blind Kriging: A New Method for Developing Metamodels , 2008 .

[50]  Kota Sridhar,et al.  コンプライアンス及び剛性楕円体を活用したコンプライアンス機構の概念的シンセシスのビルディングブロック手法 | 文献情報 | J-GLOBAL 科学技術総合リンクセンター , 2008 .

[51]  F. J. Hickernell,et al.  Uniform Experimental Designs , 2008 .

[52]  Saltelli Andrea,et al.  Global Sensitivity Analysis: The Primer , 2008 .

[53]  James O. Berger,et al.  Using Statistical and Computer Models to Quantify Volcanic Hazards , 2009, Technometrics.

[54]  Rachel T. Johnson,et al.  Design and analysis for the Gaussian process model , 2009, Qual. Reliab. Eng. Int..

[55]  Shreyes N. Melkote,et al.  Design and Analysis of Computer Experiments With Branching and Nested Factors , 2009, Technometrics.

[56]  Fabrice Gamboa,et al.  Local Polynomial Estimation for Sensitivity Analysis on Models With Correlated Inputs , 2008, Technometrics.

[57]  M. J. Bayarri,et al.  Predicting Vehicle Crashworthiness: Validation of Computer Models for Functional and Hierarchical Data , 2009 .

[58]  Michael Goldstein,et al.  Small Sample Bayesian Designs for Complex High-Dimensional Models Based on Information Gained Using Fast Approximations , 2009, Technometrics.

[59]  Wei Chen,et al.  Bayesian Validation of Computer Models , 2009, Technometrics.

[60]  Randy R. Sitter,et al.  Orthogonal and nearly orthogonal designs for computer experiments , 2009 .

[61]  Bertrand Iooss,et al.  Latin hypercube sampling with inequality constraints , 2009, 0909.0329.

[62]  Peter Z. G. Qian.  Nested Latin hypercube designs , 2009 .

[63]  Thomas J. Santner,et al.  Prediction for Computer Experiments Having Quantitative and Qualitative Input Variables , 2009, Technometrics.

[64]  Jerome Sacks,et al.  Choosing the Sample Size of a Computer Experiment: A Practical Guide , 2009, Technometrics.

[65]  Boxin Tang,et al.  Construction of orthogonal and nearly orthogonal Latin hypercubes , 2009 .

[66]  Robert G. Sargent,et al.  Verification and validation of simulation models , 2009, IEEE Engineering Management Review.

[67]  Herbert K. H. Lee,et al.  Bayesian Guided Pattern Search for Robust Local Optimization , 2009, Technometrics.

[68]  Anthony O'Hagan,et al.  Diagnostics for Gaussian Process Emulators , 2009, Technometrics.

[69]  Curtis B. Storlie,et al.  Variable Selection in Bayesian Smoothing Spline ANOVA Models: Application to Deterministic Computer Codes , 2009, Technometrics.

[70]  C. Lemieux.  Monte Carlo and Quasi-Monte Carlo Sampling , 2009 .

[71]  Thomas J. Santner,et al.  Simultaneous Determination of Tuning and Calibration Parameters for Computer Experiments , 2009, Technometrics.

[72]  Tobias Wagner,et al.  Modelling of a thermomechanically coupled forming process based on functional outputs from a finite element analysis and from experimental measurements , 2010 .

[73]  S. Lefebvre,et al.  Metamodeling of aircraft infrared signature dispersion , 2010 .

[74]  Giovanni Pistone,et al.  Comparing and generating Latin Hypercube designs in Kriging models , 2010 .

[75]  M. Ratto,et al.  Using recursive algorithms for the efficient identification of smoothing spline ANOVA models , 2010 .

[76]  A. Jourdan,et al.  Optimal Latin hypercube designs for the Kullback–Leibler criterion , 2010 .

[77]  Alistair I. Mees.  Nonlinear Dynamics and Statistics , 2012 .