PRACTICAL CONSIDERATIONS IN SELECTING THE BEST SET OF SIMULATED SYSTEMS

In many practical problems, simulation models are used to support complex decision-making processes comparing hundreds or thousands of solutions. These problems typically have a key objective but the final decision may be dependent on other factors, which cannot be incorporated into the simulation model. In such cases, decision-makers may request a short list of ‘good’ solutions, which work well for the main objective and satisfy one or more chance constraints. While fully sequential ranking and selection procedures can be effective at solving these problems, surveys of experimentation practice suggest that they are under-utilized, potentially due to difficulties automating commercial software. We develop an approach with just two stages of replications. The approach, which has been designed to cope with the use of common random numbers, draws on ideas from indifference zones and makes use of bootstrapping to find a subset of high quality solutions. A Python implementation is freely available.

[1]  R. Bechhofer.  A Single-Sample Multiple Decision Procedure for Ranking Means of Normal Populations with known Variances , 1954 .

[2]  S. Gupta.  On Some Multiple Decision (Selection and Ranking) Rules , 1965 .

[3]  A. Law,et al.  A procedure for selecting a subset of size m containing the l best of k independent normal populations, with applications to simulation , 1985 .

[4]  E. Jack Chen.  Using common random numbers for indifference-zone selection , 2001, Proceeding of the 2001 Winter Simulation Conference (Cat. No.01CH37304).

[5]  Stephen E. Chick,et al.  New Two-Stage and Sequential Procedures for Selecting the Best Simulated System , 2001, Oper. Res..

[6]  Jürgen Branke,et al.  Selecting a Selection Procedure , 2007, Manag. Sci..

[7]  Michael R Chernick,et al.  Bootstrap Methods: A Guide for Practitioners and Researchers , 2007 .

[8]  Russell C. H. Cheng,et al.  Resampling methods of analysis in simulation studies , 2009, Proceedings of the 2009 Winter Simulation Conference (WSC).

[9]  L. Lee,et al.  Finding the non-dominated Pareto set for multi-objective simulation models , 2010 .

[10]  Barry L. Nelson,et al.  Bootstrapping-based fixed-width confidence intervals for ranking and selection , 2010, Proceedings of the 2010 Winter Simulation Conference.

[11]  S. Andradóttir,et al.  Fully sequential procedures for comparing constrained systems via simulation , 2010 .

[12]  Franco Chingcuanco,et al.  A procedure to select the best subset among simulated systems using economic opportunity cost , 2013, 2013 Winter Simulations Conference (WSC).

[13]  Susan R. Hunter,et al.  Optimal Sampling Laws for Stochastically Constrained Simulation Optimization on Finite Sets , 2013, INFORMS J. Comput..

[14]  Michael C. Fu,et al.  Handbook of Simulation Optimization , 2014 .

[15]  Weiwei Chen,et al.  A note on the subset selection for simulation optimization , 2015, 2015 Winter Simulation Conference (WSC).

[16]  Jun Luo,et al.  Chance Constrained Selection of the Best , 2015, INFORMS J. Comput..

[17]  Barry L. Nelson,et al.  Computational improvements in bootstrap ranking & selection procedures via multiple comparison with the best , 2015, 2015 Winter Simulation Conference (WSC).

[18]  Weiwei Chen,et al.  A new budget allocation framework for selecting top simulated designs , 2016 .

[19]  Barry L. Nelson,et al.  General-purpose ranking and selection for computer simulation , 2016 .

[20]  Kranthi Kumar,et al.  Multi-Objective Simulation-Optimization Model for Long-term Reservoir Operation using Piecewise Linear Hedging Rule , 2018, Water Resources Management.