Scalable Constrained Bayesian Optimization

The global optimization of a high-dimensional black-box function under black-box constraints is a pervasive task in machine learning, control, and engineering. These problems are difficult since the feasible set is typically non-convex and hard to find, in addition to the curses of dimensionality and the heterogeneity of the underlying functions. In particular, these characteristics dramatically impact the performance of Bayesian optimization methods, that otherwise have become the de-facto standard for sample-efficient optimization in unconstrained settings. Due to the lack of sample-efficient methods, practitioners usually fall back to evolutionary strategies or heuristics. We propose the scalable constrained Bayesian optimization (SCBO) algorithm that addresses the above challenges by data-independent transformations of the functions and follows the recent theme of local Bayesian optimization. A comprehensive experimental evaluation demonstrates that SCBO achieves excellent results and outperforms the state-of-the-art methods.

[1]  Warren B. Powell,et al.  A unified framework for stochastic optimization , 2019, Eur. J. Oper. Res..

[2]  W. R. Thompson.  ON THE LIKELIHOOD THAT ONE UNKNOWN PROBABILITY EXCEEDS ANOTHER IN VIEW OF THE EVIDENCE OF TWO SAMPLES , 1933 .

[3]  Jakub M. Tomczak,et al.  Combinatorial Bayesian Optimization using the Graph Cartesian Product , 2019, NeurIPS.

[4]  Nando de Freitas,et al.  Taking the Human Out of the Loop: A Review of Bayesian Optimization , 2016, Proceedings of the IEEE.

[5]  Matthias Poloczek,et al.  Efficient search of compositional space for hybrid organic–inorganic perovskites via Bayesian optimization , 2018, npj Computational Materials.

[6]  Jasper Snoek,et al.  Bayesian Optimization with Unknown Constraints , 2014, UAI.

[7]  M. Powell.  A View of Algorithms for Optimization without Derivatives 1 , 2007 .

[8]  Matthias Poloczek,et al.  A Framework for Bayesian Optimization in Embedded Subspaces , 2019, ICML.

[9]  Andrew Gordon Wilson,et al.  Scalable Log Determinants for Gaussian Process Kernel Learning , 2017, NIPS.

[10]  Guilherme Ottoni,et al.  Constrained Bayesian Optimization with Noisy Experiments , 2017, Bayesian Analysis.

[11]  Helio J. C. Barbosa,et al.  Constrained Optimization Problems in Mechanical Engineering Design Using a Real-Coded Steady-State Genetic Algorithm , 2010 .

[12]  Masao Fukushima,et al.  Derivative-Free Filter Simulated Annealing Method for Constrained Continuous Global Optimization , 2006, J. Glob. Optim..

[13]  Volkan Cevher,et al.  High-Dimensional Bayesian Optimization via Additive Models with Overlapping Groups , 2018, AISTATS.

[14]  M. Powell.  A Direct Search Optimization Method That Models the Objective and Constraint Functions by Linear Interpolation , 1994 .

[15]  Jennifer G. Dy,et al.  ADMMBO: Bayesian Optimization with Unknown Constraints using ADMM , 2019, J. Mach. Learn. Res..

[16]  P. Clancy,et al.  Bypassing Solid-State Intermediates by Solvent Engineering the Crystallization Pathway in Hybrid Organic–Inorganic Perovskites , 2020 .

[17]  Zi Wang,et al.  Batched Large-scale Bayesian Optimization in High-dimensional Spaces , 2017, AISTATS.

[18]  Eric Jones,et al.  SciPy: Open Source Scientific Tools for Python , 2001 .

[19]  Matthew W. Hoffman,et al.  Predictive Entropy Search for Efficient Global Optimization of Black-box Functions , 2014, NIPS.

[20]  Andrew Gordon Wilson,et al.  Copula Processes , 2010, NIPS.

[21]  Donald R. Jones,et al.  Global versus local search in constrained optimization of computer models , 1998 .

[22]  A. Keane,et al.  EXPERIENCES WITH OPTIMIZERS IN STRUCTURAL DESIGN Adaptive Computing in Engineering Design and Control - Pymouth, , 1994 .

[23]  Victor Picheny,et al.  A Stepwise uncertainty reduction approach to constrained global optimization , 2014, AISTATS.

[24]  Sébastien Le Digabel,et al.  Modeling an Augmented Lagrangian for Blackbox Constrained Optimization , 2014, Technometrics.

[25]  Yousef Saad,et al.  Fast Estimation of tr(f(A)) via Stochastic Lanczos Quadrature , 2017, SIAM J. Matrix Anal. Appl..

[26]  James C. Spall,et al.  Introduction to Stochastic Search and Optimization. Estimation, Simulation, and Control (Spall, J.C. , 2007 .

[27]  Carlos A. Coello Coello,et al.  Constraint-handling in genetic algorithms through the use of dominance-based tournament selection , 2002, Adv. Eng. Informatics.

[28]  R. Liem,et al.  Expected drag minimization for aerodynamic design optimization based on aircraft operational data , 2017 .

[29]  R. M. Hicks,et al.  Single-Point and Multipoint Aerodynamic Shape Optimization of High-Speed Civil Transport , 2001 .

[30]  Oliver Kramer,et al.  A Review of Constraint-Handling Techniques for Evolution Strategies , 2010, Appl. Comput. Intell. Soft Comput..

[31]  David Ginsbourger,et al.  On the choice of the low-dimensional domain for global optimization via random embeddings , 2017, Journal of Global Optimization.

[32]  Andrew Gordon Wilson,et al.  Scaling Gaussian Process Regression with Derivatives , 2018, NeurIPS.

[33]  Dirk V. Arnold,et al.  A (1+1)-CMA-ES for constrained optimisation , 2012, GECCO '12.

[34]  Christine A. Shoemaker,et al.  A Stochastic Radial Basis Function Method for the Global Optimization of Expensive Functions , 2007, INFORMS J. Comput..

[35]  Peter I. Frazier,et al.  A Tutorial on Bayesian Optimization , 2018, ArXiv.

[36]  Kirthevasan Kandasamy,et al.  Parallelised Bayesian Optimisation via Thompson Sampling , 2018, AISTATS.

[37]  Nando de Freitas,et al.  Bayesian Optimization in a Billion Dimensions via Random Embeddings , 2013, J. Artif. Intell. Res..

[38]  David Ginsbourger,et al.  A Warped Kernel Improving Robustness in Bayesian Optimization Via Random Embeddings , 2014, LION.

[39]  Tim Hesterberg,et al.  Introduction to Stochastic Search and Optimization: Estimation, Simulation, and Control , 2004, Technometrics.

[40]  Matt J. Kusner,et al.  Bayesian Optimization with Inequality Constraints , 2014, ICML.

[41]  Andrew Gordon Wilson,et al.  GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration , 2018, NeurIPS.

[42]  Nikolaus Hansen,et al.  The CMA Evolution Strategy: A Comparing Review , 2006, Towards a New Evolutionary Computation.

[43]  Matthias Poloczek,et al.  Bayesian Optimization of Combinatorial Structures , 2018, ICML.

[44]  Andreas Krause,et al.  Efficient High Dimensional Bayesian Optimization with Additivity and Quadrature Fourier Features , 2018, NeurIPS.

[45]  Alán Aspuru-Guzik,et al.  Parallel and Distributed Thompson Sampling for Large-scale Accelerated Exploration of Chemical Space , 2017, ICML.

[46]  Matthew W. Hoffman,et al.  A General Framework for Constrained Bayesian Optimization using Information-based Search , 2015, J. Mach. Learn. Res..

[47]  Joaquim R. R. A. Martins,et al.  Multimission Aircraft Fuel-Burn Minimization via Multipoint Aerostructural Optimization , 2015 .

[48]  Victor Picheny,et al.  Bayesian optimization under mixed constraints with a slack-variable augmented Lagrangian , 2016, NIPS.

[49]  Max Welling,et al.  BOCK : Bayesian Optimization with Cylindrical Kernels , 2018, ICML.

[50]  Matthias Poloczek,et al.  Scalable Global Optimization via Local Bayesian Optimization , 2019, NeurIPS.