Efficient Simulation Budget Allocation for Selecting an Optimal Subset

We consider a class of the subset selection problem in ranking and selection. The objective is to identify the top m out of k designs based on simulated output. Traditional procedures are conservative and inefficient. Using the optimal computing budget allocation framework, we formulate the problem as that of maximizing the probability of correctly selecting all of the top-m designs subject to a constraint on the total number of samples available. For an approximation of this correct selection probability, we derive an asymptotically optimal allocation and propose an easy-to-implement heuristic sequential allocation procedure. Numerical experiments indicate that the resulting allocations are superior to other methods in the literature that we tested, and the relative efficiency increases for larger problems. In addition, preliminary numerical results indicate that the proposed new procedure has the potential to enhance computational efficiency for simulation optimization.

[1]  S. Gupta.  On Some Multiple Decision (Selection and Ranking) Rules , 1965 .

[2]  M. Degroot.  Optimal Statistical Decisions , 1970 .

[3]  S. Dalal,et al.  ALLOCATION OF OBSERVATIONS IN RANKING AND SELECTION WITH UNEQUAL VARIANCES , 1971 .

[4]  John H. Holland,et al.  Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence , 1992 .

[5]  Y. Rinott.  On two-stage selection procedures and related probability-inequalities , 1978 .

[6]  A. Law,et al.  A procedure for selecting a subset of size m containing the l best of k independent normal populations, with applications to simulation , 1985 .

[7]  James R. Wilson,et al.  Restricted Subset Selection Procedures for Simulation , 1989, Oper. Res..

[8]  Averill M. Law,et al.  Simulation Modeling & Analysis , 1991 .

[9]  Shumeet Baluja,et al.  A Method for Integrating Genetic Search Based Function Optimization and Competitive Learning , 1994 .

[10]  Lance D. Chambers.  Practical handbook of genetic algorithms , 1995 .

[11]  B. Nelson,et al.  Using common random numbers for indifference-zone selection and multiple comparisons in simulation , 1995 .

[12]  A. Tamhane.  Design and Analysis of Experiments for Statistical Selection, Screening, and Multiple Comparisons , 1995 .

[13]  Lance D. Chambers,et al.  Practical Handbook of Genetic Algorithms , 1995 .

[14]  Chun-Hung Chen.  A lower bound for the correct subset-selection probability and its application to discrete-event system simulations , 1996, IEEE Trans. Autom. Control..

[15]  Chun-Hung Chen,et al.  New development of optimal computing budget allocation for discrete event simulation , 1997, WSC '97.

[16]  Stephen E. Chick,et al.  Comparison of Bayesian and frequentist assessments of uncertainty for selecting the best system , 1998, 1998 Winter Simulation Conference. Proceedings (Cat. No.98CH36274).

[17]  Chun-Hung Chen,et al.  An empirical evaluation of several methods to select the best system , 1999, TOMC.

[18]  Russell C. Walker.  Introduction to Mathematical Programming , 1999 .

[19]  Russell R. Barton,et al.  Proceedings of the 2000 winter simulation conference , 2000 .

[20]  E. J. Chen,et al.  An enhanced two-stage selection procedure , 2000, 2000 Winter Simulation Conference Proceedings (Cat. No.00CH37165).

[21]  Chun-Hung Chen,et al.  Simulation Budget Allocation for Further Enhancing the Efficiency of Ordinal Optimization , 2000, Discret. Event Dyn. Syst..

[22]  Lee W. Schruben,et al.  Improved decision processes through simultaneous simulation and time dilation , 2000, 2000 Winter Simulation Conference Proceedings (Cat. No.00CH37165).

[23]  Stephen E. Chick,et al.  New Procedures to Select the Best Simulated System Using Common Random Numbers , 2001, Manag. Sci..

[24]  Stephen E. Chick,et al.  New Two-Stage and Sequential Procedures for Selecting the Best Simulated System , 2001, Oper. Res..

[25]  Barry L. Nelson,et al.  Using Ranking and Selection to "Clean Up" after Simulation Optimization , 2003, Oper. Res..

[26]  Ek Peng Chew,et al.  A simulation study on sampling and selecting under fixed computing budget , 2003, Proceedings of the 2003 Winter Simulation Conference, 2003..

[27]  Enver Yücesan,et al.  Discrete-event simulation optimization using ranking, selection, and multiple comparison procedures: A survey , 2003, TOMC.

[28]  Loo Hay Lee,et al.  Optimal computing budget allocation for multi-objective simulation models , 2004, Proceedings of the 2004 Winter Simulation Conference, 2004..

[29]  R. G. Ingalls,et al.  PROCEEDINGS OF THE 2002 WINTER SIMULATION CONFERENCE , 2002 .

[30]  Lucy Y. Pao,et al.  Computing budget allocation for efficient ranking and selection of variances with application to target tracking algorithms , 2004, IEEE Transactions on Automatic Control.

[31]  Dirk P. Kroese,et al.  The Cross-Entropy Method: A Unified Approach to Combinatorial Optimization, Monte-Carlo Simulation and Machine Learning , 2004 .

[32]  Peter Buchholz,et al.  Enhancing evolutionary algorithms with statistical selection procedures for simulation optimization , 2005, Proceedings of the Winter Simulation Conference, 2005..

[33]  Lee W. Schruben,et al.  Analysis methodology: are we done? [discrete-event simulation] , 2005, Proceedings of the Winter Simulation Conference, 2005..

[34]  Jiaqiao Hu,et al.  A Model Reference Adaptive Search Algorithm for Global Optimization , 2005 .

[35]  ENHANCING EVOLUTIONARY ALGORITHMS WITH STATISTICAL SELECTION PROCEDURES FOR SIMULATION OPTIMIZATION , 2005 .

[36]  Analysis methodology: are we done? , 2005, WSC.

[37]  Chun-Hung Chen,et al.  Efficient Dynamic Simulation Allocation in Ordinal Optimization , 2006, IEEE Transactions on Automatic Control.

[38]  S. Marcus,et al.  Model-Based Randomized Methods for Global Optimization , 2006 .

[39]  Barry L. Nelson,et al.  Comparing Systems via Simulation , 2007 .

[40]  Chun-Hung Chen,et al.  Opportunity Cost and OCBA Selection Procedures in Ordinal Optimization for a Fixed Number of Alternative Systems , 2007, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[41]  Jürgen Branke,et al.  Selecting a Selection Procedure , 2007, Manag. Sci..

[42]  Michael C. Fu,et al.  A Model Reference Adaptive Search Method for Global Optimization , 2007, Oper. Res..

[43]  Chun-Hung Chen,et al.  Simulation Allocation for Determining the Best Design in the Presence of Correlated Sampling , 2007, INFORMS J. Comput..

[44]  Michael C. Fu,et al.  A model reference adaptive search method for stochastic optimization with applications to Markov decision processes , 2007, 2007 46th IEEE Conference on Decision and Control.