Regression and Kriging metamodels with their experimental designs in simulation: A review

This article reviews the design and analysis of simulation experiments. It focusses on analysis via two types of metamodel (surrogate. emulator); namely, low-order polynomial regression, and Kriging (or Gaussian process). The metamodel type determines the design of the simulation experiment, which determines the input combinations of the simulation model. For example, a first-order polynomial regression metamodel should use a “resolution-III”design, whereas Kriging may use “Latin hypercube sampling”. More generally, polynomials of first or second order may use resolution III, IV, V, or “central composite” designs. Before applying either regression or Kriging metamodeling, the many inputs of a realistic simulation model can be screened via “sequential bifurcation”. Optimization of the simulated system may use either a sequence of low-order polynomials—known as “response surface methodology” (RSM)—or Kriging models fitted through sequential designs—including “efficient global optimization” (EGO). Finally, “robust”optimization accounts for uncertainty in some simulation inputs.

[1]  Barry L. Nelson,et al.  ‘Some tactical problems in digital simulation’ for the next 10 years , 2016, J. Simulation.

[2]  Søren Nymand Lophaven,et al.  DACE - A Matlab Kriging Toolbox, Version 2.0 , 2002 .

[3]  Margaret J. Robertson,et al.  Design and Analysis of Experiments , 2006, Handbook of statistics.

[4]  Yu Ding,et al.  Graph based isomorph-free generation of two-level regular fractional factorial designs , 2010 .

[5]  Szu Hui Ng,et al.  Calibration, Validation, and Prediction in Random Simulation Models , 2015, ACM Trans. Model. Comput. Simul..

[6]  Victor Picheny,et al.  Fast Parallel Kriging-Based Stepwise Uncertainty Reduction With Application to the Identification of an Excursion Set , 2014, Technometrics.

[7]  Jack P. C. Kleijnen,et al.  Constrained Optimization in Simulation: A Novel Approach , 2008 .

[8]  Susan M. Lewis,et al.  Design of experiments for screening , 2015, 1510.05248.

[9]  Jon C. Helton,et al.  Performance assessment for the proposed high-level radioactive waste repository at Yucca Mountain, Nevada , 2014, Reliab. Eng. Syst. Saf..

[10]  Jack P. C. Kleijnen,et al.  Kriging Metamodeling in Simulation: A Review , 2007, Eur. J. Oper. Res..

[11]  Jerome Sacks,et al.  Choosing the Sample Size of a Computer Experiment: A Practical Guide , 2009, Technometrics.

[12]  Russell R. Barton,et al.  Chapter 18 Metamodel-Based Simulation Optimization , 2006, Simulation.

[13]  Bogumil Kaminski,et al.  A method for the updating of stochastic kriging metamodels , 2015, Eur. J. Oper. Res..

[14]  Enrique del Castillo,et al.  Statitical Testing of Optimality Conditions in Multiresponse Simulation-Based Optimization , 2005, Eur. J. Oper. Res..

[15]  Augustyn Markiewicz,et al.  Optimal designs in multivariate linear models , 2007 .

[16]  Mike Preuss,et al.  High-Dimensional Model-Based Optimization Based on Noisy Evaluations of Computer Games , 2012, LION.

[17]  Russell R. Barton,et al.  A review on design, modeling and applications of computer experiments , 2006 .

[18]  Jack P. C. Kleijnen,et al.  Constrained optimization in expensive simulation: Novel approach , 2010, Eur. J. Oper. Res..

[19]  Jack P. C. Kleijnen,et al.  Robust Optimization in Simulation: Taguchi and Krige Combined , 2009, INFORMS J. Comput..

[20]  Saltelli Andrea,et al.  Global Sensitivity Analysis: The Primer , 2008 .

[21]  M. Isabel Reis dos Santos,et al.  Switching regression metamodels in stochastic simulation , 2016, Eur. J. Oper. Res..

[22]  Shuangzhe Liu,et al.  Global Sensitivity Analysis: The Primer by Andrea Saltelli, Marco Ratto, Terry Andres, Francesca Campolongo, Jessica Cariboni, Debora Gatelli, Michaela Saisana, Stefano Tarantola , 2008 .

[23]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.

[24]  Stanley H. Cohen,et al.  Design and Analysis , 2010 .

[25]  Henry W. Altland Experiments: Planning, Analysis, and Parameter Design Optimization , 2001, Technometrics.

[26]  B. Efron Frequentist accuracy of Bayesian estimates , 2015, Journal of the Royal Statistical Society. Series B, Statistical methodology.

[27]  Jack P. C. Kleijnen,et al.  Factor Screening for Simulation with Multiple Responses: Sequential Bifurcation , 2013, Eur. J. Oper. Res..

[28]  Siuli Mukhopadhyay,et al.  Response surface methodology , 2010 .

[29]  Jack P. C. Kleijnen Design and Analysis of Simulation Experiments , 2007 .

[30]  Sandeep Juneja,et al.  Nested Simulation in Portfolio Risk Measurement , 2008, Manag. Sci..

[31]  Kwon-Hee Lee,et al.  A Global Robust Optimization Using Kriging Based Approximation Model , 2006 .

[32]  Dimitri N. Mavris,et al.  Heuristics for the regression of stochastic simulations , 2013, J. Simulation.

[33]  R JonesDonald,et al.  Efficient Global Optimization of Expensive Black-Box Functions , 1998 .

[34]  M. Rendas,et al.  Extending Morris Method: identification of the interaction graph using cycle-equitabe designs , 2015 .

[35]  Murat Kulahci,et al.  Simulation-based cycle-time quantile estimation in manufacturing settings employing non-FIFO dispatching policies , 2009, J. Simulation.

[36]  Tom Dhaene,et al.  Efficient space-filling and non-collapsing sequential design strategies for simulation-based modeling , 2011, Eur. J. Oper. Res..

[37]  Hassan Maatouk,et al.  Gaussian Process Emulators for Computer Experiments with Inequality Constraints , 2016, Mathematical Geosciences.

[38]  Emanuele Borgonovo,et al.  Sensitivity analysis: A review of recent advances , 2016, Eur. J. Oper. Res..

[39]  Barry L. Nelson,et al.  Stochastic kriging for simulation metamodeling , 2008, 2008 Winter Simulation Conference.

[40]  Jack P. C. Kleijnen,et al.  An Overview of the Design and Analysis of Simulation Experiments for Sensitivity Analysis , 2005, Eur. J. Oper. Res..

[41]  Jack P. C. Kleijnen,et al.  Validation of regression metamodels in simulation: Bootstrap approach , 2006, Eur. J. Oper. Res..

[42]  Szu Hui Ng,et al.  A study on the effects of parameter estimation on kriging model's prediction error in stochastic simulations , 2009, Proceedings of the 2009 Winter Simulation Conference (WSC).

[43]  Xi Chen,et al.  Building metamodels for quantile-based measures using sectioning , 2013, 2013 Winter Simulations Conference (WSC).

[44]  Ihsan Yanikoglu,et al.  Adjustable Robust Parameter Design with Unknown Distributions , 2013 .

[45]  Jack P. C. Kleijnen,et al.  Monotonicity-preserving bootstrapped Kriging metamodels for expensive simulations , 2009, J. Oper. Res. Soc..

[46]  Jack P. C. Kleijnen,et al.  Methodology for Determining the Acceptability of Given Designs in Uncertain Environments , 2009 .

[47]  Jack P. C. Kleijnen,et al.  Customized sequential designs for random simulation experiments: Kriging metamodeling and bootstrapping , 2008, Eur. J. Oper. Res..

[48]  Kenneth Joseph Ryan,et al.  Minimum Aberration Fractional Factorial Designs With Large N , 2010, Technometrics.

[49]  Hong Wan,et al.  Combining STRONG with screening designs for large-scale simulation optimization , 2014 .

[50]  Zhenzhou Lu,et al.  Variable importance analysis: A comprehensive review , 2015, Reliab. Eng. Syst. Saf..

[51]  X. Qu Statistical properties of Rechtschaffner designs , 2007 .

[52]  Jack P. C. Kleijnen,et al.  Robust dual-response optimization , 2016 .

[53]  Barry L. Nelson,et al.  Improving the Efficiency and Efficacy of Controlled Sequential Bifurcation for Simulation Factor Screening , 2010, INFORMS J. Comput..

[54]  Bernd Bischl,et al.  Resampling Methods for Meta-Model Validation with Recommendations for Evolutionary Computation , 2012, Evolutionary Computation.

[55]  Thomas Bäck,et al.  Efficient multi-criteria optimization on noisy machine learning problems , 2015, Appl. Soft Comput..

[56]  Kwon-Hee Lee,et al.  Th World Congresses of Structural and Multidisciplinary Optimization a Global Robust Optimization Using the Kriging Based Approximation Model , 2022 .

[57]  Russell R. Barton Design of experiments for fitting subsystem metamodels , 1997, WSC '97.

[58]  Inneke Van Nieuwenhuyse,et al.  Simulation optimization in inventory replenishment: a classification , 2015 .

[59]  Huashuai Qu,et al.  Gradient Extrapolated Stochastic Kriging , 2014, TOMC.

[60]  M. E. Angun Black Box Simulation Optimization: Generalized Response Surface Methodology , 2008 .

[61]  Jack P. C. Kleijnen,et al.  Short-term robustness of production management systems: A case study , 2003, Eur. J. Oper. Res..

[62]  Tom Dhaene,et al.  On the use of gradients in Kriging surrogate models , 2014, Proceedings of the Winter Simulation Conference 2014.

[63]  Douglas C. Montgomery,et al.  Response Surface Methodology: Process and Product Optimization Using Designed Experiments , 1995 .

[64]  André I. Khuri,et al.  Response surface methodology , 2010 .

[65]  Donald R. Jones,et al.  Efficient Global Optimization of Expensive Black-Box Functions , 1998, J. Glob. Optim..

[66]  M. Rendas,et al.  Extending Morris method: identification of the interaction graph using cycle-equitable designs , 2015 .

[67]  Wei Xie,et al.  A Bayesian Framework for Quantifying Uncertainty in Stochastic Simulation , 2014, Oper. Res..

[68]  Jin Zhang,et al.  Methodology for determining the acceptability of system designs in uncertain environments , 2011, Eur. J. Oper. Res..

[69]  G. Box,et al.  On the Experimental Attainment of Optimum Conditions , 1951 .

[70]  Jean-Philippe Vial,et al.  Robust Optimization , 2021, ICORES.