GUARANTEES ON THE PROBABILITY OF GOOD SELECTION

This tutorial provides an overview of guarantees on the probability of good selection (PGS), i.e., statistical guarantees on selecting – with high probability – an alternative whose expected performance is within a given tolerance of the best. We discuss why PGS guarantees are superior to more popular, related guarantees on the probability of correct selection (PCS) under the indifference-zone formulation. We review existing procedures that deliver PGS guarantees and assess several direct and indirect methods of proof. We compare the frequentist and Bayesian interpretations of PGS and highlight the differences in how procedures are designed to deliver PGS guarantees under the two frameworks.

[1]  Jingchen Liu,et al.  Large deviations perspective on ordinal optimization of heavy-tailed systems , 2008, 2008 Winter Simulation Conference.

[2]  Peter W. Glynn,et al.  Selecting the best system, large deviations, and multi-armed bandits , 2015 .

[3]  Seong-Hee Kim,et al.  Selecting the best by comparing simulated systems in a group of three , 2011, Proceedings of the 2011 Winter Simulation Conference (WSC).

[4]  Shie Mannor,et al.  Action Elimination and Stopping Conditions for the Multi-Armed Bandit and Reinforcement Learning Problems , 2006, J. Mach. Learn. Res..

[5]  Loo Hay Lee,et al.  Ranking and Selection: Efficient Simulation Budget Allocation , 2015 .

[6]  Barry L. Nelson,et al.  A sequential procedure for neighborhood selection-of-the-best in optimization via simulation , 2006, Eur. J. Oper. Res..

[7]  Shane G. Henderson,et al.  Reusing Search Data in Ranking and Selection , 2018, ACM Trans. Model. Comput. Simul..

[8]  T. W. Anderson.  A MODIFICATION OF THE SEQUENTIAL PROBABILITY RATIO TEST TO REDUCE THE SAMPLE SIZE , 1960 .

[9]  Stephen E. Chick,et al.  New Two-Stage and Sequential Procedures for Selecting the Best Simulated System , 2001, Oper. Res..

[10]  E. Paulson.  A Sequential Procedure for Selecting the Population with the Largest Mean from $k$ Normal Populations , 1964 .

[11]  W. Fairweather.  Some extensions of Somerville's procedure for ranking means of normal populations , 1968 .

[12]  Barry L. Nelson,et al.  Indifference-Zone-Free Selection of the Best , 2016, Oper. Res..

[13]  J. Hsu.  Multiple Comparisons: Theory and Methods , 1996 .

[14]  Shie Mannor,et al.  PAC Bounds for Multi-armed Bandit and Markov Decision Processes , 2002, COLT.

[15]  Peter W. Glynn,et al.  A large deviations perspective on ordinal optimization , 2004, Proceedings of the 2004 Winter Simulation Conference, 2004..

[16]  W. David Kelton,et al.  Sequential selection procedures: Using sample means to improve efficiency , 2005, Eur. J. Oper. Res..

[17]  V. Guiard.  Different definitions of Δ-correct selection for the indifference zone formulation , 1996 .

[18]  Julie L. Swann,et al.  Simple Procedures for Selecting the Best Simulated System When the Number of Alternatives is Large , 2001, Oper. Res..

[19]  Barry L. Nelson,et al.  Selecting a good system: procedures and inference , 2001 .

[20]  Peter I. Frazier,et al.  A Fully Sequential Elimination Procedure for Indifference-Zone Ranking and Selection with Tight Bounds on Probability of Correct Selection , 2014, Oper. Res..

[21]  Milton Sobel,et al.  Selecting the best one of several binomial populations , 1957 .

[22]  I. Johnstone,et al.  ASYMPTOTICALLY OPTIMAL PROCEDURES FOR SEQUENTIAL ADAPTIVE SELECTION OF THE BEST OF SEVERAL NORMAL MEANS , 1982 .

[23]  J. Berger,et al.  A Bayesian Approach to Ranking and Selection of Related Means with Alternatives to Analysis-of-Variance Methodology , 1988 .

[24]  Y. Rinott.  On two-stage selection procedures and related probability-inequalities , 1978 .

[25]  John N. Tsitsiklis,et al.  The Sample Complexity of Exploration in the Multi-Armed Bandit Problem , 2004, J. Mach. Learn. Res..

[26]  Xi Chen,et al.  Optimal PAC Multiple Arm Identification with Applications to Crowdsourcing , 2014, ICML.

[27]  R. Bechhofer.  A Single-Sample Multiple Decision Procedure for Ranking Means of Normal Populations with known Variances , 1954 .

[28]  Rémi Munos,et al.  Pure exploration in finitely-armed and continuous-armed bandits , 2011, Theor. Comput. Sci..

[29]  Jürgen Branke,et al.  Selecting a Selection Procedure , 2007, Manag. Sci..

[30]  V. Fabian,et al.  On Multiple Decision Methods for Ranking Population Means , 1962 .

[31]  Ying Zhong,et al.  A new framework of designing sequential ranking-and-selection procedures , 2017, 2017 Winter Simulation Conference (WSC).

[32]  B. Nelson,et al.  Using common random numbers for indifference-zone selection and multiple comparisons in simulation , 1995 .

[33]  S. C. Kao,et al.  Sequential selectiok procedures based on confidence sequences for normal populations , 1980 .

[34]  A. Law,et al.  A procedure for selecting a subset of size m containing the l best of k independent normal populations, with applications to simulation , 1985 .

[35]  S. Dalal,et al.  ALLOCATION OF OBSERVATIONS IN RANKING AND SELECTION WITH UNEQUAL VARIANCES , 1971 .

[36]  S. Gupta,et al.  Bayesian look ahead one-stage sampling allocations for selection of the best population , 1996 .

[37]  Stephen E. Chick,et al.  New Procedures to Select the Best Simulated System Using Common Random Numbers , 2001, Manag. Sci..

[38]  T. W. E. Lau,et al.  Universal Alignment Probabilities and Subset Selection for Ordinal Optimization , 1997 .

[39]  Barry L. Nelson,et al.  General-purpose ranking and selection for computer simulation , 2016 .

[40]  Mark Broadie,et al.  Implications of heavy tails on simulation-based ordinal optimization , 2007, 2007 Winter Simulation Conference.

[41]  Oren Somekh,et al.  Almost Optimal Exploration in Multi-Armed Bandits , 2013, ICML.

[42]  Andrew Paul Kiekhaefer.  Simulation ranking and selection procedures and applications in network reliability design , 2011 .

[43]  Barry L. Nelson,et al.  A fully sequential procedure for indifference-zone selection in simulation , 2001, TOMC.

[44]  W. Chiu.  The ranking of means of normal populations for a generalized selection goal , 1974 .

[45]  Daniel Russo,et al.  Simple Bayesian Algorithms for Best Arm Identification , 2016, COLT.

[46]  Jürgen Branke,et al.  New developments in ranking and selection: an empirical comparison of the three main approaches , 2005, Proceedings of the Winter Simulation Conference, 2005..

[47]  I. Weissman,et al.  On the indifference zone approach to selection—A consistency result , 1981 .

[48]  Barry L. Nelson,et al.  Two-Stage Multiple Comparisons with the Best for Computer Simulation , 1995, Oper. Res..

[49]  Yaozhong Wu,et al.  Selection Procedures with Frequentist Expected Opportunity Cost Bounds , 2005, Oper. Res..

[50]  Wei-Ning Yang,et al.  A Bonferroni selection procedure when using commom random numbers with unknown variances , 1986, WSC '86.

[51]  Shane G. Henderson,et al.  An efficient fully sequential selection procedure guaranteeing probably approximately correct selection , 2017, 2017 Winter Simulation Conference (WSC).

[52]  Chun-Hung Chen,et al.  Dynamic Sampling Allocation and Design Selection , 2016, INFORMS J. Comput..

[53]  Christos G. Cassandras,et al.  Ordinal optimisation and simulation , 2000, J. Oper. Res. Soc..

[54]  Anthony J. Hayter,et al.  On the selection probabilities of two-stage decision procedures , 1994 .

[55]  Susan R. Hunter,et al.  Efficient Ranking and Selection in Parallel Computing Environments , 2015, Oper. Res..

[56]  Susan R. Hunter,et al.  Large-deviation sampling laws for constrained simulation optimization on finite sets , 2010, Proceedings of the 2010 Winter Simulation Conference.

[57]  Chun-Hung Chen,et al.  Simulation Budget Allocation for Further Enhancing the Efficiency of Ordinal Optimization , 2000, Discret. Event Dyn. Syst..

[58]  Takayuki Osogami,et al.  Finding probably best systems quickly via simulations , 2006, TOMC.