Efficient, nearly orthogonal-and-balanced, mixed designs: an effective way to conduct trade-off analyses via simulation

Designed experiments are powerful methodologies for gaining insights into the behaviour of complex simulation models. In recent years, many new designs have been created to address the large number of factors and complex response surfaces that often arise in simulation studies, but handling discrete-valued or qualitative factors remains problematic. We propose a framework for generating a design, of specified size, that is nearly orthogonal and nearly balanced for any mix of factor types (categorical, numerical discrete, and numerical continuous) and mix of factor levels. These new designs allow decision makers structured methods for trade-off analyses in situations that are not necessarily amenable to other methods for choosing alternatives, such as simulation optimization or ranking and selection approaches. These new designs also compare well to existing approaches for constructing custom designs for smaller experiments, and may also be of interest for exploring computer models in domains where fewer factors are involved.

[1]  Priya Kohli.  ORTHOGONAL ARRAYS , 2008 .

[2]  I. Sobol.  On the distribution of points in a cube and the approximate evaluation of integrals , 1967 .

[3]  Christine M. Anderson-Cook,et al.  Optimization of Designed Experiments Based on Multiple Criteria Utilizing a Pareto Frontier , 2011, Technometrics.

[4]  J. C. Wang,et al.  Nearly orthogonal arrays with mixed levels and small runs , 1992 .

[5]  Thomas W. Lucas,et al.  Designs for Large‐Scale Simulation Experiments, with Applications to Defense and Homeland Security , 2012 .

[6]  Henry P. Wynn,et al.  Maximum entropy sampling , 1987 .

[7]  Thomas W. Lucas,et al.  Efficient Nearly Orthogonal and Space-Filling Latin Hypercubes , 2007, Technometrics.

[8]  Jack P. C. Kleijnen,et al.  Robust Optimization in Simulation: Taguchi and Krige Combined , 2009, INFORMS J. Comput..

[9]  Karl Heinz Kienitz,et al.  Generating and improving orthogonal designs by using mixed integer programming , 2011, Eur. J. Oper. Res..

[10]  Jon Lee.  Maximum entropy sampling , 2001 .

[11]  Arnold J. Stromberg,et al.  Number-theoretic Methods in Statistics , 1996 .

[12]  Margaret J. Robertson,et al.  Design and Analysis of Experiments , 2006, Handbook of statistics.

[13]  Averill M. Law,et al.  Simulation Modeling and Analysis , 1982 .

[14]  M. E. Johnson,et al.  Minimax and maximin distance designs , 1990 .

[15]  Fred J. Hickernell,et al.  A generalized discrepancy and quadrature error bound , 1998, Math. Comput..

[16]  Robert E. Shannon,et al.  Design and analysis of simulation experiments , 1978, WSC '78.

[17]  Kai-Tai Fang,et al.  A new approach in constructing orthogonal and nearly orthogonal arrays , 2000 .

[18]  José Arnaldo Barra Montevechi,et al.  Sensitivity analysis in discrete-event simulation using fractional factorial designs , 2010, J. Simulation.

[19]  W. Matthew Carlyle,et al.  Constructing nearly orthogonal latin hypercubes for any nonsaturated run-variable combination , 2012, TOMC.

[20]  P. Laycock,et al.  Optimum Experimental Designs , 1995 .

[21]  Arne C. Bathke,et al.  The ANOVA F test can still be used in some balanced designs with unequal variances and nonnormal data , 2004 .

[22]  Douglas C. Montgomery,et al.  Response Surface Methodology: Process and Product Optimization Using Designed Experiments , 1995 .

[23]  Lih-Yuan Deng,et al.  Orthogonal Arrays: Theory and Applications , 1999, Technometrics.

[24]  W. Näther.  Optimum experimental designs , 1994 .

[25]  M. Kendall.  Statistical Methods for Research Workers , 1937, Nature.

[26]  Karl Heinz Kienitz,et al.  Improved efficient, nearly orthogonal, nearly balanced mixed designs , 2011, Proceedings of the 2011 Winter Simulation Conference (WSC).

[27]  Jack P. C. Kleijnen,et al.  State-of-the-Art Review: A User's Guide to the Brave New World of Designing Simulation Experiments , 2005, INFORMS J. Comput..

[28]  Henry P. Wynn,et al.  [Design and Analysis of Computer Experiments]: Rejoinder , 1989 .

[29]  Sonja Kuhnt,et al.  Design and analysis of computer experiments , 2010 .

[30]  Hongquan Xu,et al.  An Algorithm for Constructing Orthogonal and Nearly-Orthogonal Arrays With Mixed Levels and Small Runs , 2002, Technometrics.

[31]  Thomas P. Ryan,et al.  Modern Regression Methods , 1996 .

[32]  G. Box,et al.  Some New Three Level Designs for the Study of Quantitative Variables , 1960 .

[33]  Aleš Florian,et al.  An efficient sampling scheme: Updated Latin Hypercube Sampling , 1992 .

[34]  Esther Seiden,et al.  On Orthogonal Arrays , 1966 .

[35]  Jack P. C. Kleijnen,et al.  Simulation experiments in practice: statistical design and regression analysis , 2007, J. Simulation.

[36]  Hong Wan,et al.  Work smarter, not harder: A tutorial on designing and conducting simulation experiments , 2015, 2015 Winter Simulation Conference (WSC).

[37]  V. R. Joseph,et al.  ORTHOGONAL-MAXIMIN LATIN HYPERCUBE DESIGNS , 2008 .

[38]  Jason Liu,et al.  PrimoGENI for hybrid network simulation and emulation experiments in GENI , 2012, J. Simulation.

[39]  Min-Qian Liu,et al.  An algorithmic approach to constructing mixed-level orthogonal and near-orthogonal arrays , 2008, Comput. Stat. Data Anal..