Discrete Convex Simulation Optimization

We propose new sequential simulation algorithms for general convex simulation optimization problems with high-dimensional discrete decision space. The performance of each discrete decision variable is evaluated via stochastic simulation replications. Our proposed simulation algorithms utilize the discrete convex structure and are guaranteed with high probability to find a solution that is close to the best within any given user-specified precision level. The proposed algorithms work for any general convex problem and the efficiency is demonstrated by proved upper bounds on simulation costs. The upper bounds demonstrate a polynomial dependence on the dimension and scale of the decision space. For some discrete simulation optimization problems, a gradient estimator may be available at low costs along with a single simulation replication. By integrating gradient estimators, which are possibly biased, we propose simulation algorithms to achieve optimality guarantees with a reduced dependence on the dimension under moderate assumptions on the bias.

[1]  Saeed Ghadimi,et al.  Stochastic First- and Zeroth-Order Methods for Nonconvex Stochastic Programming , 2013, SIAM J. Optim..

[2]  John N. Tsitsiklis,et al.  The Sample Complexity of Exploration in the Multi-Armed Bandit Problem , 2004, J. Mach. Learn. Res..

[3]  R. Bechhofer A Single-Sample Multiple Decision Procedure for Ranking Means of Normal Populations with known Variances , 1954 .

[4]  David B. Shmoys,et al.  Minimizing Multimodular Functions and Allocating Capacity in Bike‐Sharing Systems , 2016, IPCO.

[5]  Eitan Altman,et al.  Discrete-Event Control of Stochastic Networks - Multimodularity and Regularity , 2004, Lecture notes in mathematics.

[6]  Matthew Malloy,et al.  lil' UCB : An Optimal Exploration Algorithm for Multi-Armed Bandits , 2013, COLT.

[7]  Shinji Ito,et al.  Submodular Function Minimization with Noisy Evaluation Oracle , 2019, NeurIPS.

[8]  Barry L. Nelson,et al.  Discrete Optimization via Simulation , 2015 .

[9]  Shane G. Henderson,et al.  An efficient fully sequential selection procedure guaranteeing probably approximately correct selection , 2017, 2017 Winter Simulation Conference (WSC).

[10]  Honggang Wang,et al.  Integer-Ordered Simulation Optimization using R-SPLINE: Retrospective Search with Piecewise-Linear Interpolation and Neighborhood Enumeration , 2013, TOMC.

[11]  Aurélien Garivier,et al.  On the Complexity of A/B Testing , 2014, COLT.

[12]  Elad Hazan,et al.  An optimal algorithm for stochastic strongly-convex optimization , 2010, 1006.2425.

[13]  Seong-Hee Kim,et al.  Designing an optimal water quality monitoring network for river systems using constrained discrete optimization via simulation , 2014 .

[14]  Barry L. Nelson,et al.  Rapid Discrete Optimization via Simulation with Gaussian Markov Random Fields , 2020, INFORMS J. Comput..

[15]  Nisheeth K. Vishnoi,et al.  Convex Optimization with Unbounded Nonconvex Oracles using Simulated Annealing , 2018, COLT.

[16]  Warren B. Powell,et al.  The Knowledge-Gradient Policy for Correlated Normal Beliefs , 2009, INFORMS J. Comput..

[17]  Yurii Nesterov,et al.  Random Gradient-Free Minimization of Convex Functions , 2015, Foundations of Computational Mathematics.

[18]  Clément Bouttier,et al.  Convergence Rate of a Simulated Annealing Algorithm with Noisy Observations , 2017, J. Mach. Learn. Res..

[19]  Barry L. Nelson,et al.  Gaussian Markov Random Fields for Discrete Optimization via Simulation: Framework and Algorithms , 2019, Oper. Res..

[20]  藤重 悟 Submodular functions and optimization , 1991 .

[21]  Eunji Lim,et al.  Stochastic approximation over multidimensional discrete sets with applications to inventory systems and admission control of queueing networks , 2012, TOMC.

[22]  Dmitriy Drusvyatskiy,et al.  Pathological Subgradient Dynamics , 2019, SIAM J. Optim..

[23]  Georg Ch. Pflug,et al.  Simulated Annealing for noisy cost functions , 1996, J. Glob. Optim..

[24]  Jian Li,et al.  Towards Instance Optimal Bounds for Best Arm Identification , 2016, COLT.

[25]  Wouter M. Koolen,et al.  Non-Asymptotic Pure Exploration by Solving Games , 2019, NeurIPS.

[26]  Suvrit Sra,et al.  On Complexity of Finding Stationary Points of Nonsmooth Nonconvex Functions , 2020, ArXiv.

[27]  Sebastian U. Stich,et al.  Analysis of SGD with Biased Gradient Estimators , 2020, ArXiv.

[28]  Di Wu,et al.  Analyzing and provably improving fixed budget ranking and selection algorithms , 2018, ArXiv.

[29]  Susan R. Hunter,et al.  Parallel Ranking and Selection , 2017 .

[30]  Shane G. Henderson,et al.  Simulation optimization for a large-scale bike-sharing system , 2016, 2016 Winter Simulation Conference (WSC).

[31]  Barry L. Nelson,et al.  Optimization via simulation over discrete decision variables , 2010 .

[32]  Xi Chen,et al.  Enhancing Stochastic Kriging Metamodels with Gradient Estimators , 2013, Oper. Res..

[33]  Wouter M. Koolen,et al.  Pure Exploration with Multiple Correct Answers , 2019, NeurIPS.

[34]  Per Ola Börjesson,et al.  Simple Approximations of the Error Function Q(x) for Communications Applications , 1979, IEEE Trans. Commun..

[35]  P. Favati Convexity in nonlinear integer programming , 1990 .

[36]  Oren Somekh,et al.  Almost Optimal Exploration in Multi-Armed Bandits , 2013, ICML.

[37]  Tianbao Yang,et al.  RSG: Beating Subgradient Method without Smoothness and Strong Convexity , 2015, J. Mach. Learn. Res..

[38]  Tianbao Yang,et al.  Accelerated Stochastic Subgradient Methods under Local Error Bound Condition , 2016 .

[39]  SHUBHADA,et al.  Optimal $δ$-Correct Best-Arm Selection for Heavy-Tailed Distributions , 2020, ALT.

[40]  Jiaqiao Hu,et al.  A Model Reference Adaptive Search Method for Stochastic Global Optimization , 2008, Commun. Inf. Syst..

[41]  Peter Stone,et al.  Efficient Selection of Multiple Bandit Arms: Theory and Practice , 2010, ICML.

[42]  Shivaram Kalyanakrishnan,et al.  Information Complexity in Bandit Subset Selection , 2013, COLT.

[43]  Michael R. Lyu,et al.  Pure Exploration of Multi-Armed Bandits with Heavy-Tailed Payoffs , 2018, UAI.

[44]  Barry L. Nelson,et al.  A fully sequential procedure for indifference-zone selection in simulation , 2001, TOMC.

[45]  Huashuai Qu,et al.  Gradient Extrapolated Stochastic Kriging , 2014, TOMC.

[46]  Aurélien Garivier,et al.  On the Complexity of Best-Arm Identification in Multi-Armed Bandit Models , 2014, J. Mach. Learn. Res..

[47]  Alexander Shapiro,et al.  The Sample Average Approximation Method for Stochastic Discrete Optimization , 2002, SIAM J. Optim..

[48]  Dmitriy Drusvyatskiy,et al.  Stochastic subgradient method converges at the rate O(k-1/4) on weakly convex functions , 2018, ArXiv.

[49]  Brian Axelrod,et al.  Near-optimal Approximate Discrete and Continuous Submodular Function Minimization , 2019, SODA.

[50]  Peter I. Frazier,et al.  A Fully Sequential Elimination Procedure for Indifference-Zone Ranking and Selection with Tight Bounds on Probability of Correct Selection , 2014, Oper. Res..

[51]  L. Jeff Hong,et al.  Review on Ranking and Selection: A New Perspective , 2020 .

[52]  Kazuo Murota,et al.  Discrete convex analysis , 1998, Math. Program..

[53]  Flagot Yohannes Derivative free optimization methods , 2012 .

[54]  Shie Mannor,et al.  Action Elimination and Stopping Conditions for the Multi-Armed Bandit and Reinforcement Learning Problems , 2006, J. Mach. Learn. Res..

[55]  Shane G. Henderson,et al.  GUARANTEES ON THE PROBABILITY OF GOOD SELECTION , 2018, 2018 Winter Simulation Conference (WSC).

[56]  Michael I. Jordan,et al.  On the Local Minima of the Empirical Risk , 2018, NeurIPS.

[57]  René David,et al.  Discrete event dynamic systems , 1989 .

[58]  Aurélien Garivier,et al.  Optimal Best Arm Identification with Fixed Confidence , 2016, COLT.

[59]  Ambuj Tewari,et al.  PAC Subset Selection in Stochastic Multi-armed Bandits , 2012, ICML.

[60]  John Darzentas,et al.  Problem Complexity and Method Efficiency in Optimization , 1983 .

[61]  Sham M. Kakade,et al.  Stochastic Convex Optimization with Bandit Feedback , 2011, SIAM J. Optim..

[62]  Jie Xu,et al.  Industrial strength COMPASS: A comprehensive algorithm and software for optimization via simulation , 2010, TOMC.

[63]  Shane G. Henderson,et al.  Predicting the Simulation Budget in Ranking and Selection Procedures , 2019, ACM Trans. Model. Comput. Simul..

[64]  Chun-Hung Chen A lower bound for the correct subset-selection probability and its application to discrete-event system simulations , 1996, IEEE Trans. Autom. Control..

[65]  A. Shapiro,et al.  The Sample Average Approximation Method for Stochastic Programs with Integer Recourse , 2002 .

[66]  Satoru Fujishige Theory of submodular programs: A fenchel-type min-max theorem and subgradients of submodular functions , 1984, Math. Program..

[67]  S. Matthew Weinberg,et al.  New Query Lower Bounds for Submodular Function MInimization , 2020, ITCS.

[68]  J. Shanthikumar,et al.  Stochastic convexity and its applications , 1988, Advances in Applied Probability.

[69]  J. Burkey,et al.  WEAK SHARP MINIMA IN MATHEMATICAL PROGRAMMING , 1993 .

[70]  Mikael Johansson,et al.  Convergence of a Stochastic Gradient Method with Momentum for Nonsmooth Nonconvex Optimization , 2020, ICML.

[71]  Daniel Russo,et al.  Simple Bayesian Algorithms for Best Arm Identification , 2016, COLT.

[72]  R. Wolff,et al.  On the convexity of loss probabilities , 2002, Journal of Applied Probability.

[73]  R. Munos,et al.  Best Arm Identification in Multi-Armed Bandits , 2010, COLT.

[74]  Barry L. Nelson,et al.  Discrete Optimization via Simulation Using COMPASS , 2006, Oper. Res..

[75]  Tianbao Yang,et al.  Accelerate Stochastic Subgradient Method by Leveraging Local Error Bound , 2016, ArXiv.

[76]  Michael C. Fu,et al.  A Model Reference Adaptive Search Method for Global Optimization , 2007, Oper. Res..

[77]  Rémi Munos,et al.  Pure exploration in finitely-armed and continuous-armed bandits , 2011, Theor. Comput. Sci..

[78]  Chun-Hung Chen,et al.  Ranking and Selection as Stochastic Control , 2017, IEEE Transactions on Automatic Control.

[79]  Seong-Hee Kim,et al.  Penalty Function with Memory for Discrete Optimization via Simulation with Stochastic Constraints , 2015, Oper. Res..

[80]  Huashuai Qu,et al.  Regression Models Augmented with Direct Stochastic Gradient Estimators , 2014, INFORMS J. Comput..

[81]  Hariharan Narayanan,et al.  Escaping the Local Minima via Simulated Annealing: Optimization of Approximately Convex Functions , 2015, COLT.

[82]  Chun-Hung Chen,et al.  Simulation Budget Allocation for Further Enhancing the Efficiency of Ordinal Optimization , 2000, Discret. Event Dyn. Syst..

[83]  Shie Mannor,et al.  PAC Bounds for Multi-armed Bandit and Markov Decision Processes , 2002, COLT.

[84]  B. Feng,et al.  BIASED GRADIENT ESTIMATORS IN SIMULATION OPTIMIZATION , 2020 .

[85]  Yurii Nesterov,et al.  Lectures on Convex Optimization , 2018 .

[86]  Jie Chen,et al.  Stochastic Gradient Descent with Biased but Consistent Gradient Estimators , 2018, ArXiv.

[87]  Diego Klabjan,et al.  Improving the Expected Improvement Algorithm , 2017, NIPS.

[88]  Ohad Shamir A Variant of Azuma's Inequality for Martingales with Subgaussian Tails , 2011, ArXiv.

[89]  Seong-Hee Kim,et al.  Comparison with a standard via fully sequential procedures , 2005, TOMC.

[90]  Yin Tat Lee,et al.  A Faster Cutting Plane Method and its Implications for Combinatorial and Convex Optimization , 2015, 2015 IEEE 56th Annual Symposium on Foundations of Computer Science.

[91]  Yurii Nesterov,et al.  First-order methods of smooth convex optimization with inexact oracle , 2013, Mathematical Programming.

[92]  Alexandra Carpentier,et al.  Tight (Lower) Bounds for the Fixed Budget Best Arm Identification Bandit Problem , 2016, COLT.

[93]  Yifan Hu,et al.  Biased Stochastic Gradient Descent for Conditional Stochastic Optimization , 2020, ArXiv.

[94]  Jiaqiao Hu,et al.  Discrete optimization via gradient-based adaptive stochastic search methods , 2018 .

[95]  Dmitriy Drusvyatskiy,et al.  Stochastic Subgradient Method Converges on Tame Functions , 2018, Foundations of Computational Mathematics.

[96]  Barry L. Nelson,et al.  Indifference-Zone-Free Selection of the Best , 2016, Oper. Res..

[97]  Martin J. Wainwright,et al.  Optimal Rates for Zero-Order Convex Optimization: The Power of Two Function Evaluations , 2013, IEEE Transactions on Information Theory.

[98]  Loo Hay Lee,et al.  Stochastic Simulation Optimization - An Optimal Computing Budget Allocation , 2010, System Engineering and Operations Research.

[99]  Martin Zinkevich,et al.  Online Convex Programming and Generalized Infinitesimal Gradient Ascent , 2003, ICML.

[100]  Jun Luo,et al.  Fully Sequential Procedures for Large-Scale Ranking-and-Selection Problems in Parallel Computing Environments , 2015, Oper. Res..

[101]  G. Pflug,et al.  Optimal allocation of simulation experiments in discrete stochastic optimization and approximative algorithms , 1997 .

[102]  Tito Homem-de-Mello,et al.  Monte Carlo sampling-based methods for stochastic optimization , 2014 .

[103]  P. L’Ecuyer,et al.  A Unified View of the IPA, SF, and LR Gradient Estimation Techniques , 1990 .

[104]  Stephen E. Chick,et al.  New Two-Stage and Sequential Procedures for Selecting the Best Simulated System , 2001, Oper. Res..

[105]  Jie Xu,et al.  Speeding up COMPASS for high-dimensional discrete optimization via simulation , 2010, Oper. Res. Lett..

[106]  Julia L. Higle,et al.  Stabilization of Cutting Plane Algorithms for Stochastic Linear Programming Problems , 2009, Encyclopedia of Optimization.

[107]  Leyuan Shi,et al.  A New Budget Allocation Framework for the Expected Opportunity Cost , 2017, Oper. Res..

[108]  Georg Ch. Pflug,et al.  Confidence sets for discrete stochastic optimization , 1995, Ann. Oper. Res..

[109]  Niao He,et al.  On the Convergence Rate of Stochastic Mirror Descent for Nonsmooth Nonconvex Optimization , 2018, 1806.04781.

[110]  Francis Bach,et al.  Submodular functions: from discrete to continuous domains , 2015, Mathematical Programming.

[111]  David B. Shmoys,et al.  Predicting Bike Usage for New York City's Bike Sharing System , 2015, AAAI Workshop: Computational Sustainability.

[112]  Lihua Sun,et al.  Balancing Exploitation and Exploration in Discrete Optimization via Simulation Through a Gaussian Process-Based Search , 2014, Oper. Res..

[113]  Krishnakumar Balasubramanian,et al.  Zeroth-Order Nonconvex Stochastic Optimization: Handling Constraints, High Dimensionality, and Saddle Points , 2018, Foundations of Computational Mathematics.