A survey of adaptive sampling for global metamodeling in support of simulation-based complex engineering design

Metamodeling is becoming a rather popular means to approximate the expensive simulations in today’s complex engineering design problems since accurate metamodels can bring in a lot of benefits. The metamodel accuracy, however, heavily depends on the locations of the observed points. Adaptive sampling, as its name suggests, places more points in regions of interest by learning the information from previous data and metamodels. Consequently, compared to traditional space-filling sampling approaches, adaptive sampling has great potential to build more accurate metamodels with fewer points (simulations), thereby gaining increasing attention and interest by both practitioners and academicians in various fields. Noticing that there is a lack of reviews on adaptive sampling for global metamodeling in the literature, which is needed, this article categorizes, reviews, and analyzes the state-of-the-art single−/multi-response adaptive sampling approaches for global metamodeling in support of simulation-based engineering design. In addition, we also review and discuss some important issues that affect the success of an adaptive sampling approach as well as providing brief remarks on adaptive sampling for other purposes. Last, challenges and future research directions are provided and discussed.

[1]  Wei Tian,et al.  A review of sensitivity analysis methods in building energy analysis , 2013 .

[2]  Naif Alajlan,et al.  Active learning for spectroscopic data regression , 2012 .

[3]  Bin Li,et al.  Accurate and efficient processor performance prediction via regression tree based modeling , 2009, J. Syst. Archit..

[4]  G. Gary Wang,et al.  Survey of modeling and optimization strategies to solve high-dimensional design problems with computationally-expensive black-box functions , 2010 .

[5]  Weichung Wang,et al.  Discrete particle swarm optimization for constructing uniform design on irregular regions , 2014, Comput. Stat. Data Anal..

[6]  Alípio Mário Jorge,et al.  Ensemble approaches for regression: A survey , 2012, CSUR.

[7]  Saltelli Andrea,et al.  Global Sensitivity Analysis: The Primer , 2008 .

[8]  Kenny Q. Ye,et al.  Algorithmic construction of optimal symmetric Latin hypercube designs , 2000 .

[9]  Anders Krogh,et al.  Neural Network Ensembles, Cross Validation, and Active Learning , 1994, NIPS.

[10]  K. Yamazaki,et al.  Sequential Approximate Optimization using Radial Basis Function network for engineering optimization , 2011 .

[11]  Jason L. Loeppky,et al.  Batch sequential designs for computer experiments , 2010 .

[12]  Haitao Liu,et al.  A Robust Error-Pursuing Sequential Sampling Approach for Global Metamodeling Based on Voronoi Diagram and Cross Validation , 2014 .

[13]  Jack P. C. Kleijnen,et al.  Application-driven sequential designs for simulation experiments: Kriging metamodelling , 2004, J. Oper. Res. Soc..

[14]  Andy J. Keane,et al.  On the Design of Optimization Strategies Based on Global Response Surface Approximation Models , 2005, J. Glob. Optim..

[15]  Selen Cremaschi,et al.  Adaptive sequential sampling for surrogate model generation with artificial neural networks , 2014, Comput. Chem. Eng..

[16]  Boxin Tang,et al.  NESTED SPACE-FILLING DESIGNS FOR COMPUTER EXPERIMENTS WITH TWO LEVELS OF ACCURACY , 2009 .

[17]  Stefan Görtz,et al.  A Variable-Fidelity Modeling Method for Aero-Loads Prediction , 2010 .

[18]  Thomas J. Santner,et al.  The Design and Analysis of Computer Experiments , 2003, Springer Series in Statistics.

[19]  Alberto Lovison,et al.  Adaptive sampling with a Lipschitz criterion for accurate metamodeling , 2010 .

[20]  Neil D. Lawrence,et al.  Kernels for Vector-Valued Functions: a Review , 2011, Found. Trends Mach. Learn..

[21]  Andy J. Keane,et al.  Review of efficient surrogate infill sampling criteria with constraint handling , 2010 .

[22]  D Deschrijver,et al.  Adaptive Sampling Algorithm for Macromodeling of Parameterized $S$ -Parameter Responses , 2011, IEEE Transactions on Microwave Theory and Techniques.

[23]  V. R. Joseph,et al.  ORTHOGONAL-MAXIMIN LATIN HYPERCUBE DESIGNS , 2008 .

[24]  W. J. Studden,et al.  Theory Of Optimal Experiments , 1972 .

[25]  Caroline Sainvitu,et al.  Adaptive infill sampling criterion for multi-fidelity optimization based on Gappy-POD , 2016 .

[26]  Wei Chen,et al.  Optimizing Latin hypercube design for sequential sampling of computer experiments , 2009 .

[27]  T. Simpson,et al.  Comparative studies of metamodelling techniques under multiple modelling criteria , 2001 .

[28]  Jon Louis Bentley,et al.  Quad trees a data structure for retrieval on composite keys , 1974, Acta Informatica.

[29]  Shengli Xu,et al.  Constrained global optimization via a DIRECT-type constraint-handling technique and an adaptive metamodeling strategy , 2017 .

[30]  Jon C. Helton,et al.  A comparison of uncertainty and sensitivity analysis results obtained with random and Latin hypercube sampling , 2005, Reliab. Eng. Syst. Saf..

[31]  Yong Zhang,et al.  Uniform Design: Theory and Application , 2000, Technometrics.

[32]  G. G. Wang,et al.  Mode-pursuing sampling method for global optimization on expensive black-box functions , 2004 .

[33]  Xin Wei,et al.  A new sequential optimal sampling method for radial basis functions , 2012, Appl. Math. Comput..

[34]  Liang Gao,et al.  An adaptive SVR-HDMR model for approximating high dimensional problems , 2015 .

[35]  Tom Dhaene,et al.  Efficient space-filling and non-collapsing sequential design strategies for simulation-based modeling , 2011, Eur. J. Oper. Res..

[36]  Massimiliano Vasile,et al.  Computational methods in engineering design and optimization , 2013 .

[37]  Dick den Hertog,et al.  Constrained Maximin Designs for Computer Experiments , 2003, Technometrics.

[38]  Anirban Chaudhuri,et al.  Parallel surrogate-assisted global optimization with expensive functions – a survey , 2016 .

[39]  Robert B. Gramacy,et al.  Adaptive design of supercomputer experiments , 2008 .

[40]  Nicholas R. Jennings,et al.  Real-time information processing of environmental sensor network data using bayesian gaussian processes , 2012, ACM Trans. Sens. Networks.

[41]  Andrea Grosso,et al.  Finding maximin latin hypercube designs by Iterated Local Search heuristics , 2009, Eur. J. Oper. Res..

[42]  Morteza Haghighat Sefat,et al.  Development of an adaptive surrogate model for production optimization , 2015 .

[43]  Jason L. Loeppky,et al.  Batch sequential design to achieve predictive maturity with calibrated computer models , 2011, Reliab. Eng. Syst. Saf..

[44]  Fabien Teytaud,et al.  A Survey of Meta-heuristics Used for Computing Maximin Latin Hypercube , 2014, EvoCOP.

[45]  Peter Z. G. Qian,et al.  An Approach to Constructing Nested Space-Filling Designs for Multi-Fidelity Computer Experiments. , 2010, Statistica Sinica.

[46]  Shengli Xu,et al.  Optimal Weighted Pointwise Ensemble of Radial Basis Functions with Different Basis Functions , 2016 .

[47]  Jerome Sacks,et al.  Choosing the Sample Size of a Computer Experiment: A Practical Guide , 2009, Technometrics.

[48]  C. D. Perttunen,et al.  Lipschitzian optimization without the Lipschitz constant , 1993 .

[49]  Ali Ajdari,et al.  An Adaptive Exploration-Exploitation Algorithm for Constructing Metamodels in Random Simulation Using a Novel Sequential Experimental Design , 2014, Commun. Stat. Simul. Comput..

[50]  Peng Wang,et al.  A Novel Latin Hypercube Algorithm via Translational Propagation , 2014, TheScientificWorldJournal.

[51]  Russell R. Barton.  Design of experiments for fitting subsystem metamodels , 1997, WSC '97.

[52]  M. E. Johnson,et al.  Minimax and maximin distance designs , 1990 .

[53]  H. Sebastian Seung,et al.  Query by committee , 1992, COLT '92.

[54]  Shapour Azarm,et al.  Bayesian meta‐modelling of engineering design simulations: a sequential approach with adaptation to irregularities in the response behaviour , 2005 .

[55]  Pierre Goovaerts,et al.  Adaptive Experimental Design Applied to Ergonomics Testing Procedure , 2002, DAC 2002.

[56]  Edwin R. van Dam,et al.  Bounds for Maximin Latin Hypercube Designs , 2007, Oper. Res..

[57]  Masoud Rais-Rohani,et al.  Ensemble of Metamodels with Optimized Weight Factors , 2008 .

[58]  Dick den Hertog,et al.  Maximin Latin Hypercube Designs in Two Dimensions , 2007, Oper. Res..

[59]  Shapour Azarm,et al.  An accumulative error based adaptive design of experiments for offline metamodeling , 2009 .

[60]  Bryan A. Tolson,et al.  Review of surrogate modeling in water resources , 2012 .

[61]  Bertrand Iooss,et al.  Numerical studies of space-filling designs: optimization of Latin Hypercube Samples and subprojection properties , 2013, J. Simulation.

[62]  Ying Ma,et al.  An Adaptive Bayesian Sequential Sampling Approach for Global Metamodeling , 2016 .

[63]  S. Sundararajan,et al.  Predictive Approaches for Choosing Hyperparameters in Gaussian Processes , 1999, Neural Computation.

[64]  G. G. Wang,et al.  Metamodeling for High Dimensional Simulation-Based Design Problems , 2010 .

[65]  G. Venter,et al.  An algorithm for fast optimal Latin hypercube design of experiments , 2010 .

[66]  Tom Dhaene,et al.  Sequential design and rational metamodelling , 2005, Proceedings of the Winter Simulation Conference, 2005..

[67]  S. Azarm,et al.  Approximation of multiresponse deterministic engineering simulations: a dependent metamodeling approach , 2006 .

[68]  Loïc Le Gratiet,et al.  Cokriging-Based Sequential Design Strategies Using Fast Cross-Validation Techniques for Multi-Fidelity Computer Codes , 2015, Technometrics.

[69]  Edwin V. Bonilla,et al.  Multi-task Gaussian Process Prediction , 2007, NIPS.

[70]  T. Simpson,et al.  Computationally Inexpensive Metamodel Assessment Strategies , 2002 .

[71]  Shmuel Rippa,et al.  An algorithm for selecting a good value for the parameter c in radial basis function interpolation , 1999, Adv. Comput. Math..

[72]  Jack P. C. Kleijnen,et al.  Kriging Metamodels and Their Designs , 2015 .

[73]  William I. Notz,et al.  Sequential adaptive designs in computer experiments for response surface model fit , 2008 .

[74]  Andreas Krause,et al.  Information-Theoretic Regret Bounds for Gaussian Process Optimization in the Bandit Setting , 2009, IEEE Transactions on Information Theory.

[75]  Haitao Liu,et al.  Generalized Radial Basis Function-Based High-Dimensional Model Representation Handling Existing Random Data , 2017 .

[76]  Feng Qian,et al.  Adaptive Sampling for Surrogate Modelling with Artificial Neural Network and its Application in an Industrial Cracking Furnace , 2016 .

[77]  Yao Lin,et al.  An Efficient Robust Concept Exploration Method and Sequential Exploratory Experimental Design , 2004 .

[78]  G. G. Wang,et al.  Adaptive Response Surface Method Using Inherited Latin Hypercube Design Points , 2003 .

[79]  Dong-Hoon Choi,et al.  Construction of the radial basis function based on a sequential sampling approach using cross-validation , 2009 .

[80]  Wen Yao,et al.  A gradient-based sequential radial basis function neural network modeling method , 2009, Neural Computing and Applications.

[81]  Qiang Yang,et al.  A Survey on Transfer Learning , 2010, IEEE Transactions on Knowledge and Data Engineering.

[82]  Robert B. Gramacy,et al.  Ja n 20 08 Bayesian Treed Gaussian Process Models with an Application to Computer Modeling , 2009 .

[83]  Rahul Rai,et al.  Q2S2: A New Methodology for Merging Quantitative and Qualitative Information in Experimental Design , 2008 .

[84]  Grigorios Tsoumakas,et al.  Multi-target regression via input space expansion: treating targets as inputs , 2012, Machine Learning.

[85]  G. Rennen,et al.  Nested maximin Latin hypercube designs , 2009 .

[86]  Haitao Liu,et al.  A multi-response adaptive sampling approach for global metamodeling , 2018 .

[87]  Robert E. Shannon,et al.  Design and analysis of simulation experiments , 1978, WSC '78.

[88]  F. Viana.  Things you wanted to know about the Latin hypercube design and were afraid to ask , 2016 .

[89]  Loic Le Gratiet,et al.  RECURSIVE CO-KRIGING MODEL FOR DESIGN OF COMPUTER EXPERIMENTS WITH MULTIPLE LEVELS OF FIDELITY , 2012, 1210.0686.

[90]  Xinyu Shao,et al.  A variable fidelity information fusion method based on radial basis function , 2017, Adv. Eng. Informatics.

[91]  Tom Dhaene,et al.  Adaptive CAD-model building algorithm for general planar microwave structures , 1999 .

[92]  Genetha A. Gray,et al.  Sequential design for achieving estimated accuracy of global sensitivities , 2013 .

[93]  Richard J. Beckman,et al.  A Comparison of Three Methods for Selecting Values of Input Variables in the Analysis of Output From a Computer Code , 2000, Technometrics.

[94]  Cameron J. Turner,et al.  A Review and Evaluation of Existing Adaptive Sampling Criteria and Methods for the Creation of NURBs-Based Metamodels , 2011 .

[95]  Youngdeok Hwang,et al.  Asymmetric Nested Lattice Samples , 2014, Technometrics.

[96]  Daniel Busby,et al.  Hierarchical adaptive experimental design for Gaussian process emulators , 2009, Reliab. Eng. Syst. Saf..

[97]  Christian B Allen,et al.  Investigation of an adaptive sampling method for data interpolation using radial basis functions , 2010 .

[98]  Jack P. C. Kleijnen.  Design and Analysis of Simulation Experiments , 2007 .

[99]  Eric Petit,et al.  ASK: Adaptive Sampling Kit for Performance Characterization , 2012, Euro-Par.

[100]  Urmila M. Diwekar,et al.  An efficient sampling technique for off-line quality control , 1997 .

[101]  Christian B Allen,et al.  Aerodynamic Data Modeling using Multi-Criteria Adaptive Sampling , 2010 .

[102]  N. M. Alexandrov,et al.  A trust-region framework for managing the use of approximation models in optimization , 1997 .

[103]  I. Sobol.  On the Systematic Search in a Hypercube , 1979 .

[104]  Alexander I. J. Forrester,et al.  Multi-fidelity optimization via surrogate modelling , 2007, Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[105]  S. Rippa,et al.  Numerical Procedures for Surface Fitting of Scattered Data by Radial Functions , 1986 .

[106]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.

[107]  G. Gary Wang,et al.  Review of Metamodeling Techniques in Support of Engineering Design Optimization , 2007, DAC 2006.

[108]  Shengli Xu,et al.  Sequential sampling designs based on space reduction , 2015 .

[109]  Luc Pronzato,et al.  Design of computer experiments: space filling and beyond , 2011, Statistics and Computing.

[110]  S. T. Buckland,et al.  An Introduction to the Bootstrap. , 1994 .

[111]  Zhong-Hua Han,et al.  A New Cokriging Method for Variable-Fidelity Surrogate Modeling of Aerodynamic Data , 2010 .

[112]  Liang Gao,et al.  An enhanced RBF-HDMR integrated with an adaptive sampling method for approximating high dimensional problems in engineering design , 2016 .

[113]  Udo Hahn,et al.  Multi-Task Active Learning for Linguistic Annotations , 2008, ACL.

[114]  David J. J. Toal,et al.  Some considerations regarding the use of multi-fidelity Kriging in the construction of surrogate models , 2015 .

[115]  Raphael T. Haftka,et al.  Remarks on multi-fidelity surrogates , 2016, Structural and Multidisciplinary Optimization.

[116]  Liang Zhao,et al.  World Congress on Structural and Multidisciplinary Optimization Sequential-sampling-based Kriging Method with Dynamic Basis Selection , 2022 .

[117]  Eric Walter,et al.  An informational approach to the global optimization of expensive-to-evaluate functions , 2006, J. Glob. Optim..

[118]  Thomas J. Santner,et al.  Noncollapsing Space-Filling Designs for Bounded Nonrectangular Regions , 2012, Technometrics.

[119]  Bin Li,et al.  A survey on instance selection for active learning , 2012, Knowledge and Information Systems.

[120]  Xinyu Shao,et al.  A novel sequential exploration-exploitation sampling strategy for global metamodeling , 2015 .

[121]  G. Johannesson,et al.  Comparison of Sequential Designs of Computer Experiments in High Dimensions , 2011 .

[122]  Loic Le Gratiet,et al.  Metamodel-based sensitivity analysis: polynomial chaos expansions and Gaussian processes , 2016, 1606.04273.

[123]  Dennis K. J. Lin,et al.  Computer Experiments With Both Qualitative and Quantitative Variables , 2016, Technometrics.

[124]  Raphael T. Haftka,et al.  Surrogate-based Analysis and Optimization , 2005 .

[125]  P. Sagaut,et al.  Towards an adaptive POD/SVD surrogate model for aeronautic design , 2011 .

[126]  Stefan Görtz,et al.  Improving variable-fidelity surrogate modeling via gradient-enhanced kriging and a generalized hybrid bridge function , 2013 .

[127]  Songqing Shan,et al.  Turning Black-Box Functions Into White Functions , 2011 .

[128]  Serge Guillas,et al.  Sequential Design with Mutual Information for Computer Experiments (MICE): Emulation of a Tsunami Model , 2014, SIAM/ASA J. Uncertain. Quantification.

[129]  R. Haftka,et al.  Review of multi-fidelity models , 2016, Advances in Computational Science and Engineering.

[130]  T. Therneau,et al.  An Introduction to Recursive Partitioning Using the RPART Routines , 2015 .

[131]  R. Haftka,et al.  Multiple surrogates: how cross-validation errors can help us to obtain the best predictor , 2009 .

[132]  M. D. McKay,et al.  A comparison of three methods for selecting values of input variables in the analysis of output from a computer code , 2000 .

[133]  Gérard Dreyfus,et al.  Towards the Optimal Design of Numerical Experiments , 2008, IEEE Transactions on Neural Networks.

[134]  Henry P. Wynn,et al.  Maximum entropy sampling , 1987 .

[135]  Christine A. Shoemaker,et al.  A Stochastic Radial Basis Function Method for the Global Optimization of Expensive Functions , 2007, INFORMS J. Comput..

[136]  Christine A. Shoemaker,et al.  Constrained Global Optimization of Expensive Black Box Functions Using Radial Basis Functions , 2005, J. Glob. Optim..

[137]  Yaochu Jin,et al.  Surrogate-assisted evolutionary computation: Recent advances and future challenges , 2011, Swarm Evol. Comput..

[138]  Jack P. C. Kleijnen,et al.  An Overview of the Design and Analysis of Simulation Experiments for Sensitivity Analysis , 2005, Eur. J. Oper. Res..

[139]  Robert B. Gramacy,et al.  Adaptive Design and Analysis of Supercomputer Experiments , 2008, Technometrics.

[140]  Peng Wang,et al.  A Sequential Optimization Sampling Method for Metamodels with Radial Basis Functions , 2014, TheScientificWorldJournal.

[141]  Yu Gao,et al.  A Memetic Differential Evolutionary Algorithm for High Dimensional Functions' Optimization , 2007, Third International Conference on Natural Computation (ICNC 2007).

[142]  Leonard G. C. Hamey,et al.  Minimisation of data collection by active learning , 1995, Proceedings of ICNN'95 - International Conference on Neural Networks.

[143]  Cristina H. Amon,et al.  On Adaptive Sampling for Single and Multi-Response Bayesian Surrogate Models , 2006, DAC 2006.

[144]  H. D. Patterson.  The Errors of Lattice Sampling , 1954 .

[145]  Benjamin Peherstorfer,et al.  Survey of multifidelity methods in uncertainty propagation, inference, and optimization , 2018, SIAM Rev..

[146]  Sonja Kuhnt,et al.  Design and analysis of computer experiments , 2010 .

[147]  Xin Yao,et al.  Differential evolution for high-dimensional function optimization , 2007, 2007 IEEE Congress on Evolutionary Computation.

[148]  Nantiwat Pholdee,et al.  An efficient optimum Latin hypercube sampling technique based on sequencing optimisation using simulated annealing , 2015, Int. J. Syst. Sci..

[149]  Dick den Hertog,et al.  Nested Maximin Latin Hypercube Designs in Two Dimensions , 2005 .

[150]  Ruichen Jin,et al.  On Sequential Sampling for Global Metamodeling in Engineering Design , 2002, DAC 2002.

[151]  Elie Bienenstock,et al.  Neural Networks and the Bias/Variance Dilemma , 1992, Neural Computation.

[152]  D. Shahsavani,et al.  An adaptive design and interpolation technique for extracting highly nonlinear response surfaces from deterministic models , 2009, Reliab. Eng. Syst. Saf..

[153]  S. Dreyfus,et al.  Thermodynamical Approach to the Traveling Salesman Problem : An Efficient Simulation Algorithm , 2004 .

[154]  Mohan S. Kankanhalli,et al.  Near-Optimal Active Learning of Multi-Output Gaussian Processes , 2015, AAAI.

[155]  Qi-Jun Zhang,et al.  Neural Network Training-Driven Adaptive Sampling Algorithm for Microwave Modeling , 2000, 2000 30th European Microwave Conference.

[156]  Hui Zhou,et al.  An active learning variable-fidelity metamodelling approach based on ensemble of metamodels and objective-oriented sequential sampling , 2016 .

[157]  Cristina H. Amon,et al.  Multiresponse Metamodeling in Simulation-Based Design Applications , 2012 .

[158]  Franz Aurenhammer,et al.  Voronoi diagrams—a survey of a fundamental geometric data structure , 1991, CSUR.

[159]  Brian J. Williams,et al.  Batch sequential design of optimal experiments for improved predictive maturity in physics-based modeling , 2013 .

[160]  Burr Settles,et al.  Active Learning Literature Survey , 2009 .

[161]  J. Halton.  On the efficiency of certain quasi-random sequences of points in evaluating multi-dimensional integrals , 1960 .

[162]  B. Iooss,et al.  A Review on Global Sensitivity Analysis Methods , 2014, 1404.2405.

[163]  Tom Dhaene,et al.  A Fuzzy Hybrid Sequential Design Strategy for Global Surrogate Modeling of High-Dimensional Computer Experiments , 2015, SIAM J. Sci. Comput..

[164]  M. Liefvendahl,et al.  A study on algorithms for optimization of Latin hypercubes , 2006 .

[165]  Timothy W. Simpson,et al.  Metamodeling in Multidisciplinary Design Optimization: How Far Have We Really Come? , 2014 .

[166]  Caroline Sainvitu,et al.  Adaptive sampling strategies for non‐intrusive POD‐based surrogates , 2013 .

[167]  Andy J. Keane,et al.  Recent advances in surrogate-based optimization , 2009 .

[168]  Hamdy Amin Elseramegy.  CART (Classification and Regression Trees) Program: The Implementation of the CART Program and Its Application to Estimating Attrition Rates. , 1985 .

[169]  Zhijian Wu,et al.  Enhanced opposition-based differential evolution for solving high-dimensional continuous optimization problems , 2011, Soft Comput..

[170]  Peter Z. G. Qian,et al.  Bayesian Hierarchical Modeling for Integrating Low-Accuracy and High-Accuracy Experiments , 2008, Technometrics.

[171]  Joaquim R. R. A. Martins,et al.  Multidisciplinary design optimization: A survey of architectures , 2013 .

[172]  Aaron Quan.  Batch Sequencing Methods for Computer Experiments , 2014 .

[173]  Victor Picheny,et al.  Using Cross Validation to Design Conservative Surrogates , 2010 .

[174]  R. Stocki,et al.  A method to improve design reliability using optimal Latin hypercube sampling , 2005 .

[175]  Ross D. King,et al.  Active Learning for Regression Based on Query by Committee , 2007, IDEAL.

[176]  Michael James Sasena,et al.  Flexibility and efficiency enhancements for constrained global design optimization with kriging approximations. , 2002 .

[177]  Wataru Yamazaki,et al.  A Dynamic Sampling Method for Kriging and Cokriging Surrogate Models , 2011 .

[178]  A. O'Hagan,et al.  Predicting the output from a complex computer code when fast approximations are available , 2000 .

[179]  Tom Dhaene,et al.  A balanced sequential design strategy for global surrogate modeling , 2013, 2013 Winter Simulations Conference (WSC).

[180]  Matthew W. Hoffman,et al.  Predictive Entropy Search for Efficient Global Optimization of Black-box Functions , 2014, NIPS.

[181]  Bryan A. Tolson,et al.  Numerical assessment of metamodelling strategies in computationally intensive optimization , 2012, Environ. Model. Softw..

[182]  Dirk Gorissen,et al.  A Novel Hybrid Sequential Design Strategy for Global Surrogate Modeling of Computer Experiments , 2011, SIAM J. Sci. Comput..

[183]  Peter Z. G. Qian.  Nested Latin hypercube designs , 2009 .

[184]  Erdem Acar,et al.  Simultaneous optimization of shape parameters and weight factors in ensemble of radial basis functions , 2014 .

[185]  Wei Chen,et al.  An Efficient Algorithm for Constructing Optimal Design of Computer Experiments , 2005, DAC 2003.

[186]  Robert Tibshirani,et al.  An Introduction to the Bootstrap , 1994 .

[187]  Christian B Allen,et al.  Comparison of Adaptive Sampling Methods for Generation of Surrogate Aerodynamic Models , 2013 .

[188]  Piet Demeester,et al.  A Surrogate Modeling and Adaptive Sampling Toolbox for Computer Based Design , 2010, J. Mach. Learn. Res..

[189]  V. Schulz,et al.  Comparing sampling strategies for aerodynamic Kriging surrogate models , 2012 .

[190]  Richard H. Crawford,et al.  Multidimensional sequential sampling for NURBs-based metamodel development , 2007, Engineering with Computers.

[191]  H. Sebastian Seung,et al.  Information, Prediction, and Query by Committee , 1992, NIPS.

[192]  Emanuele Borgonovo,et al.  Sensitivity analysis: A review of recent advances , 2016, Eur. J. Oper. Res..

[193]  A. O'Hagan,et al.  Bayesian calibration of computer models , 2001 .

[194]  Z. Dong,et al.  Metamodelling and search using space exploration and unimodal region elimination for design optimization , 2010 .

[195]  Jason Weston,et al.  A unified architecture for natural language processing: deep neural networks with multitask learning , 2008, ICML '08.

[196]  Timothy W. Simpson,et al.  Analysis of support vector regression for approximation of complex engineering analyses , 2003, DAC 2003.

[197]  Susan M. Sanchez,et al.  WORK SMARTER, NOT HARDER: A TUTORIAL ON DESIGNING AND CONDUCTING SIMULATION EXPERIMENTS , 2018, 2018 Winter Simulation Conference (WSC).

[198]  Peter Z. G. Qian,et al.  Nested Lattice Sampling: A New Sampling Scheme Derived by Randomizing Nested Orthogonal Arrays , 2010 .

[199]  Jack P. C. Kleijnen,et al.  Kriging Metamodeling in Simulation: A Review , 2007, Eur. J. Oper. Res..

[200]  Wei Chen,et al.  A non‐stationary covariance‐based Kriging method for metamodelling in engineering design , 2007 .

[201]  Abdur Rouf,et al.  Unified measurement system with suction control for measuring hysteresis in soil‐gas transport parameters , 2012 .

[202]  X. Liu,et al.  Modeling Multiresponse Surfaces for Airfoil Design with Multiple-Output-Gaussian-Process Regression , 2014 .

[203]  Jason L. Loeppky,et al.  Projection array based designs for computer experiments , 2012 .

[204]  Mehdi Ghoreyshi,et al.  Accelerating the Numerical Generation of Aerodynamic Models for Flight Simulation , 2009 .

[205]  Farrokh Mistree,et al.  A Sequential Exploratory Experimental Design Method: Development of Appropriate Empirical Models in Design , 2004, DAC 2004.

[206]  Rich Caruana,et al.  Learning Many Related Tasks at the Same Time with Backpropagation , 1994, NIPS.

[207]  Karel Crombecq,et al.  Surrogate modeling of computer experiments with sequential experimental design , 2011 .

[208]  Yang Song,et al.  A global optimization algorithm for simulation-based problems via the extended DIRECT scheme , 2015 .

[209]  Dick den Hertog,et al.  One-dimensional nested maximin designs , 2010, J. Glob. Optim..

[210]  Jon C. Helton,et al.  Sensitivity analysis in conjunction with evidence theory representations of epistemic uncertainty , 2006, Reliab. Eng. Syst. Saf..

[211]  J. Burkardt,et al.  LATINIZED, IMPROVED LHS, AND CVT POINT SETS IN HYPERCUBES , 2007 .

[212]  N. Cressie.  Spatial prediction and ordinary kriging , 1988 .

[213]  Vikrant Chandramohan Aute,et al.  Single and Multiresponse Adaptive Design of Experiments with Application to Design Optimization of Novel Heat Exchangers , 2009 .

[214]  Neil D. Lawrence,et al.  Sparse Convolved Gaussian Processes for Multi-output Regression , 2008, NIPS.

[215]  Armin Iske,et al.  Hierarchical Nonlinear Approximation for Experimental Design and Statistical Data Fitting , 2007, SIAM J. Sci. Comput..

[216]  D. Steinberg.  CART: Classification and Regression Trees , 2009 .

[217]  Donald R. Jones,et al.  Efficient Global Optimization of Expensive Black-Box Functions , 1998, J. Glob. Optim..

[218]  Peter Z. G. Qian,et al.  Gaussian Process Models for Computer Experiments With Qualitative and Quantitative Factors , 2008, Technometrics.

[219]  Hans-Martin Gutmann,et al.  A Radial Basis Function Method for Global Optimization , 2001, J. Glob. Optim..

[220]  S. Chuanga,et al.  Uniform design over general input domains with applications to target region estimation in computer experiments , 2009 .

[221]  Boxin Tang,et al.  Construction of orthogonal and nearly orthogonal Latin hypercubes , 2009 .

[222]  T. J. Mitchell,et al.  Bayesian design and analysis of computer experiments: Use of derivatives in surface prediction , 1993 .

[223]  Yves Auffray,et al.  Maximin design on non hypercube domains and kernel interpolation , 2010, Stat. Comput..

[224]  M. Isabel Reis dos Santos,et al.  Sequential experimental designs for nonlinear regression metamodels in simulation , 2008, Simul. Model. Pract. Theory.

[225]  Anna Kucerová,et al.  Competitive Comparison of Optimal Designs of Experiments for Sampling-based Sensitivity Analysis , 2012, ArXiv.

[226]  Reinhard Radermacher,et al.  Cross-validation based single response adaptive design of experiments for Kriging metamodeling of deterministic computer simulations , 2013 .

[227]  Guangyao Li,et al.  High dimensional model representation (HDMR) coupled intelligent sampling strategy for nonlinear problems , 2012, Comput. Phys. Commun..