Design and Analysis of Computational Experiments: Overview

This chapter presents an overview of the design and analysis of computational experiments with optimization algorithms. It covers classic designs and their corresponding (meta)models; namely, Resolution-III designs including fractional factorial two-level designs for first-order polynomial models, Resolution-IV and Resolution-V designs for two-factor interactions, and designs including central composite designs for second-degree polynomials. It also reviews factor screening in experiments with very many factors, focusing on the sequential bifurcation method. Furthermore, it reviews Kriging models and their designs. Finally, it discusses experiments aimed at the optimization of the parameters of a given optimization algorithm, allowing multiple random experimental outputs. This optimization may use either generalized response surface methodology or Kriging combined with mathematical programming; the discussion also covers Taguchian robust optimization.

[1]  Barry L. Nelson,et al.  Stochastic kriging for simulation metamodeling , 2008, WSC 2008.

[2]  Steven G. Gilmour,et al.  Factor Screening via Supersaturated Designs , 2006 .

[3]  Noel A Cressie,et al.  Statistics for Spatial Data, Revised Edition. , 1994 .

[4]  Douglas C. Montgomery,et al.  Response Surface Methodology: Process and Product Optimization Using Designed Experiments , 1995 .

[5]  Thomas Bartz-Beielstein,et al.  the future of experimental research , 2009, GECCO '09.

[6]  C. F. Jeff Wu,et al.  Experiments: Planning, Analysis, and Parameter Design Optimization , 2000 .

[7]  Jack P. C. Kleijnen,et al.  The correct Kriging variance estimated by bootstrapping , 2006, J. Oper. Res. Soc..

[8]  Jack P. C. Kleijnen,et al.  Screening for the important factors in large discrete-event simulation : Sequential bifurcation and its applications , 2006 .

[9]  Thomas Bartz-Beielstein,et al.  Sequential Parameter Optimization (SPO) and the role of tuning in experimental analysis , 2006 .

[10]  Søren Nymand Lophaven,et al.  DACE - A Matlab Kriging Toolbox, Version 2.0 , 2002 .

[11]  Enrique del Castillo,et al.  Statitical Testing of Optimality Conditions in Multiresponse Simulation-Based Optimization , 2005, Eur. J. Oper. Res..

[12]  Daniel Kudenko,et al.  Tuning an Algorithm Using Design of Experiments , 2010, Experimental Methods for the Analysis of Optimization Algorithms.

[13]  Jack P. C. Kleijnen,et al.  State-of-the-Art Review: A User's Guide to the Brave New World of Designing Simulation Experiments , 2005, INFORMS J. Comput..

[14]  Jack P. C. Kleijnen,et al.  Searching for important factors in simulation models with many factors: Sequential bifurcation , 1997 .

[15]  Jack P. C. Kleijnen.  Design and Analysis of Simulation Experiments , 2007 .

[16]  Jack P. C. Kleijnen,et al.  Customized sequential designs for random simulation experiments: Kriging metamodeling and bootstrapping , 2008, Eur. J. Oper. Res..

[17]  Hong-Fwu Yu,et al.  Designing a screening experiment with a reciprocal Weibull degradation rate , 2007, Comput. Ind. Eng..

[18]  Hong Wan,et al.  Controlled sequential bifurcation for software reliability study , 2007, 2007 Winter Simulation Conference.

[19]  Jack P. C. Kleijnen,et al.  Constrained optimization in expensive simulation: Novel approach , 2010, Eur. J. Oper. Res..

[20]  Jack P. C. Kleijnen,et al.  Optimization of simulated systems: OptQuest and alternatives , 2007, Simul. Model. Pract. Theory.

[21]  Gül Gürkan,et al.  Response surface methodology with stochastic constraints for expensive simulation , 2009, J. Oper. Res. Soc..

[22]  Arkadi Nemirovski,et al.  Robust optimization – methodology and applications , 2002, Math. Program..

[23]  Daniel Kudenko,et al.  Screening the parameters affecting heuristic performance , 2007, GECCO '07.

[24]  G. Box,et al.  On the Experimental Attainment of Optimum Conditions , 1951 .

[25]  M. Fu.  What you should know about simulation and derivatives , 2008 .

[26]  Jack P. C. Kleijnen,et al.  Robust Optimization in Simulation: Taguchi and Response Surface Methodology , 2008 .

[27]  William P. Gardiner,et al.  11 – Response Surface Methods , 1998 .

[28]  Jack P. C. Kleijnen,et al.  A methodology for fitting and validating metamodels in simulation , 2000, Eur. J. Oper. Res..

[29]  Manuel Laguna,et al.  Fine-Tuning of Algorithms Using Fractional Experimental Designs and Local Search , 2006, Oper. Res..

[30]  Søren Nymand Lophaven,et al.  DACE - A Matlab Kriging Toolbox , 2002 .

[31]  Thomas Bartz-Beielstein,et al.  Experimental Research in Evolutionary Computation - The New Experimentalism , 2010, Natural Computing Series.

[32]  Cem Saydam,et al.  Developing effective meta-heuristics for a probabilistic location model via experimental design , 2007, Eur. J. Oper. Res..

[33]  K. Ng,et al.  Kriging model with modified nugget effect for random simulation with heterogeneous variances , 2008, 2008 IEEE International Conference on Industrial Engineering and Engineering Management.

[34]  Jack P. C. Kleijnen,et al.  Kriging for interpolation in random simulation , 2003, J. Oper. Res. Soc..

[35]  Jack P. C. Kleijnen,et al.  Robust simulation-optimization using metamodels , 2009, Proceedings of the 2009 Winter Simulation Conference (WSC).

[36]  Jack P. C. Kleijnen,et al.  Constrained Optimization in Simulation: A Novel Approach , 2008 .

[37]  Jack P. C. Kleijnen,et al.  Finding the Important Factors in Large Discrete-Event Simulation: Sequential Bifurcation and its Applications , 2003 .

[38]  Averill M. Law,et al.  Simulation Modeling and Analysis , 1982 .

[39]  Gerald T. Mackulak,et al.  D-Optimal Sequential Experiments for Generating a Simulation-Based Cycle Time-Throughput Curve , 2002, Oper. Res..

[40]  B. Leupen,et al.  Design and analysis , 1997 .

[41]  S. T. Buckland,et al.  An Introduction to the Bootstrap. , 1994 .

[42]  Jack P. C. Kleijnen,et al.  Response surface methodology's steepest ascent and step size revisited: Correction , 2006, Eur. J. Oper. Res..

[43]  Thomas J. Santner,et al.  The Design and Analysis of Computer Experiments , 2003, Springer Series in Statistics.