Dynamic Sampling Allocation and Design Selection

We formulate the statistical selection problem in a general dynamic framework comprising fully sequential sampling allocation and optimal design selection. Because the traditional probability of correct selection measure is not sufficient to capture both aspects in this more general framework, we introduce the integrated probability of correct selection to better characterize the objective. As a result, the usual selection policy of choosing the design with the largest sample mean as the estimate of the best is no longer necessarily optimal. Rather, the optimal selection policy is to choose the design that maximizes the posterior integrated probability of correct selection, which is a function of the posterior mean and the correlation structure induced by the posterior variance. Because determining the optimal selection policy is generally intractable, we also devise an approximation scheme to efficiently approximate the optimal selection policy. For the allocation policy, we study an asymptotic policy ca...

[1]  M. Degroot.  Optimal Statistical Decisions , 1970 .

[2]  Carl M. Harris,et al.  Fundamentals of queueing theory , 1975 .

[3]  Averill M. Law,et al.  Simulation Modeling and Analysis , 1982 .

[4]  J. Berger,et al.  A Bayesian Approach to Ranking and Selection of Related Means with Alternatives to Analysis-of-Variance Methodology , 1988 .

[5]  S. Gupta,et al.  Bayesian look ahead one-stage sampling allocations for selection of the best population , 1996 .

[6]  Stephen E. Chick,et al.  Comparison of Bayesian and frequentist assessments of uncertainty for selecting the best system , 1998, 1998 Winter Simulation Conference. Proceedings (Cat. No.98CH36274).

[7]  Chun-Hung Chen,et al.  An empirical evaluation of several methods to select the best system , 1999, TOMC.

[8]  Chun-Hung Chen,et al.  Simulation Budget Allocation for Further Enhancing the Efficiency of Ordinal Optimization , 2000, Discret. Event Dyn. Syst..

[9]  Stephen E. Chick,et al.  New Two-Stage and Sequential Procedures for Selecting the Best Simulated System , 2001, Oper. Res..

[10]  Chun-Hung Chen,et al.  Optimal computing budget allocation for Monte Carlo simulation with application to product design , 2003, Simul. Model. Pract. Theory.

[11]  Peter W. Glynn,et al.  A large deviations perspective on ordinal optimization , 2004, Proceedings of the 2004 Winter Simulation Conference, 2004..

[12]  Galin L. Jones.  On the Markov chain central limit theorem , 2004, math/0409112.

[13]  Chun-Hung Chen,et al.  Intelligent simulation for alternatives comparison and application to air traffic management , 2005 .

[14]  Chun-Hung Chen,et al.  Efficient Dynamic Simulation Allocation in Ordinal Optimization , 2006, IEEE Transactions on Automatic Control.

[15]  Barry L. Nelson,et al.  Comparing Systems via Simulation , 2007 .

[16]  Chun-Hung Chen,et al.  Opportunity Cost and OCBA Selection Procedures in Ordinal Optimization for a Fixed Number of Alternative Systems , 2007, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[17]  Jürgen Branke,et al.  Selecting a Selection Procedure , 2007, Manag. Sci..

[18]  Chun-Hung Chen,et al.  Simulation Allocation for Determining the Best Design in the Presence of Correlated Sampling , 2007, INFORMS J. Comput..

[19]  Warren B. Powell,et al.  A Knowledge-Gradient Policy for Sequential Information Collection , 2008, SIAM J. Control. Optim..

[20]  Stephen E. Chick,et al.  Economic Analysis of Simulation Selection Problems , 2009, Manag. Sci..

[21]  Jürgen Branke,et al.  Sequential Sampling to Myopically Maximize the Expected Value of Information , 2010, INFORMS J. Comput..

[22]  Huashuai Qu,et al.  Ranking and selection with unknown correlation structures , 2012, Proceedings Title: Proceedings of the 2012 Winter Simulation Conference (WSC).

[23]  P. Frazier,et al.  Tutorial: Optimization via simulation with Bayesian statistics and dynamic programming , 2012, Proceedings Title: Proceedings of the 2012 Winter Simulation Conference (WSC).

[24]  Warrren B Powell,et al.  Ranking and Selection , 2012 .

[25]  Chun-Hung Chen,et al.  A dynamic framework for statistical selection problems , 2013, 2013 Winter Simulations Conference (WSC).

[26]  Yijie Peng,et al.  Efficient Simulation Resource Sharing and Allocation for Selecting the Best , 2013, IEEE Transactions on Automatic Control.

[27]  Loo Hay Lee,et al.  Stochastically Constrained Ranking and Selection via SCORE , 2014, ACM Trans. Model. Comput. Simul..