On the Convergence Rates of Expected Improvement Methods

We consider a ranking and selection problem with independent normal observations, and we analyze the asymptotic sampling rates of expected improvement (EI) methods in this setting. Such methods often perform well in practice, but a tractable analysis of their convergence rates is difficult because of the nonlinearity and nonconvexity of the EI calculations. We present new results indicating that, for known sampling noise, variants of EI produce asymptotic simulation allocations that are essentially identical to those chosen by the optimal computing budget allocation (OCBA) methodology, which is known to yield near-optimal asymptotic performance in ranking and selection. This is the first general equivalence result between EI and OCBA, and it provides insight into the good practical performance of EI. We also derive the limiting allocation for EI under unknown sampling variance.

[1]  Adam D. Bull,et al.  Convergence Rates of Efficient Global Optimization Algorithms , 2011, J. Mach. Learn. Res..

[2]  Benjamin Van Roy,et al.  Learning to Optimize via Posterior Sampling , 2013, Math. Oper. Res..

[3]  Chun-Hung Chen,et al.  Opportunity Cost and OCBA Selection Procedures in Ordinal Optimization for a Fixed Number of Alternative Systems , 2007, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[4]  Éva Tardos,et al.  Algorithm design , 2005 .

[5]  Warren B. Powell,et al.  Finite-time Analysis for the Knowledge-Gradient Policy , 2016, SIAM J. Control. Optim..

[6]  Stephen E. Chick,et al.  New Two-Stage and Sequential Procedures for Selecting the Best Simulated System , 2001, Oper. Res..

[7]  Barry L. Nelson,et al.  A brief introduction to optimization via simulation , 2009, Proceedings of the 2009 Winter Simulation Conference (WSC).

[8]  Jürgen Branke,et al.  Selecting a Selection Procedure , 2007, Manag. Sci..

[9]  Peter Auer,et al.  Finite-time Analysis of the Multiarmed Bandit Problem , 2002, Machine Learning.

[10]  Warren B. Powell,et al.  The Knowledge-Gradient Policy for Correlated Normal Beliefs , 2009, INFORMS J. Comput..

[11]  Thorsten Dickhaus,et al.  Asymptotic Tail Properties of Student's t-Distribution , 2008 .

[12]  Andrew P. Soms.  Rational Bounds for the t-Tail Area , 1980 .

[13]  Stephen E. Chick,et al.  Economic Analysis of Simulation Selection Problems , 2009, Manag. Sci..

[14]  Peter W. Glynn,et al.  Ordinal optimization: A nonparametric framework , 2011, Proceedings of the 2011 Winter Simulation Conference (WSC).

[15]  Ilya O. Ryzhov.  Expected improvement is equivalent to OCBA , 2015, 2015 Winter Simulation Conference (WSC).

[16]  Stephen E. Chick,et al.  Chapter 9 Subjective Probability and Bayesian Methodology , 2006, Simulation.

[17]  S. Gupta,et al.  Bayesian look ahead one-stage sampling allocations for selection of the best population , 1996 .

[18]  Warren B. Powell,et al.  A Knowledge-Gradient Policy for Sequential Information Collection , 2008, SIAM J. Control. Optim..

[19]  Warren B. Powell,et al.  Information Collection on a Graph , 2011, Oper. Res..

[20]  Loo Hay Lee,et al.  Stochastically Constrained Ranking and Selection via SCORE , 2014, ACM Trans. Model. Comput. Simul..

[21]  Leyuan Shi,et al.  An optimal opportunity cost selection procedure for a fixed number of designs , 2014, Proceedings of the Winter Simulation Conference 2014.

[22]  Huashuai Qu,et al.  Simulation optimization: A tutorial overview and recent developments in gradient-based methods , 2014, Proceedings of the Winter Simulation Conference 2014.

[23]  Barry L. Nelson,et al.  Recent advances in ranking and selection , 2007, 2007 Winter Simulation Conference.

[24]  Boris Defourny,et al.  Efficient learning of donor retention strategies for the American Red Cross , 2013, 2013 Winter Simulations Conference (WSC).

[25]  Loo Hay Lee,et al.  Stochastic Simulation Optimization - An Optimal Computing Budget Allocation , 2010, System Engineering and Operations Research.

[26]  Ilya O. Ryzhov,et al.  Optimal learning with non-Gaussian rewards , 2013, 2013 Winter Simulations Conference (WSC).

[27]  Loo Hay Lee,et al.  Ranking and Selection: Efficient Simulation Budget Allocation , 2015 .

[28]  T. L. Lai Andherbertrobbins.  Asymptotically Efficient Adaptive Allocation Rules , 1985 .

[29]  Andrew P. Soms,et al.  An Asymptotic Expansion for the Tail Area of the t -Distribution , 1976 .

[30]  Loo Hay Lee,et al.  Efficient Simulation Budget Allocation for Selecting an Optimal Subset , 2008, INFORMS J. Comput..

[31]  Warren B. Powell,et al.  The Knowledge Gradient Algorithm for a General Class of Online Learning Problems , 2012, Oper. Res..

[32]  Huashuai Qu,et al.  Sequential Selection with Unknown Correlation Structures , 2015, Oper. Res..

[33]  M. Degroot.  Optimal Statistical Decisions , 1970 .

[34]  Peter W. Glynn,et al.  A large deviations perspective on ordinal optimization , 2004, Proceedings of the 2004 Winter Simulation Conference, 2004..

[35]  Chun-Hung Chen,et al.  Simulation Allocation for Determining the Best Design in the Presence of Correlated Sampling , 2007, INFORMS J. Comput..

[36]  Warren B. Powell,et al.  Paradoxes in Learning and the Marginal Value of Information , 2010, Decis. Anal..

[37]  Avraham Adler,et al.  Lambert-W Function , 2015 .

[38]  Donald R. Jones,et al.  Efficient Global Optimization of Expensive Black-Box Functions , 1998, J. Glob. Optim..

[39]  Loo Hay Lee,et al.  Efficient simulation budget allocation with regression , 2013 .

[40]  Chun-Hung Chen,et al.  Simulation Budget Allocation for Further Enhancing the Efficiency of Ordinal Optimization , 2000, Discret. Event Dyn. Syst..

[41]  Jürgen Branke,et al.  Sequential Sampling to Myopically Maximize the Expected Value of Information , 2010, INFORMS J. Comput..

[42]  Warren B. Powell,et al.  Optimal Learning: Powell/Optimal , 2012 .

[43]  S. Jain,et al.  CALIBRATING SIMULATION MODELS USING THE KNOWLEDGE GRADIENT WITH CONTINUOUS PARAMETERS , 2010 .