pBO-2GP-3B: A batch parallel known/unknown constrained Bayesian optimization with feasibility classification and its applications in computational fluid dynamics

Abstract In this work, we present a constrained batch-parallel Bayesian optimization (BO) framework, termed pBO-2GP-3B, to accelerate the optimization process for high-dimensional and computationally expensive problems, with known and unknown constraints. Two Gaussian processes (GPs) are simultaneously constructed: one models the objective function, whereas the other models the unknown constraints. The known constraint is penalized directly into the acquisition function. For every iteration, three batches are built in sequential order: the first two are the acquisition hallucination and the exploration batches for the objective GP, respectively, and the third one is the exploration batch for the classification GP. The pBO-2GP-3B optimization framework is demonstrated with three synthetic examples (2D and 6D), as well as a 33D multi-phase solid–liquid computational fluid dynamics (CFD) model for the design optimization of a centrifugal slurry pump impeller.

[1]  Søren Nymand Lophaven,et al.  Aspects of the Matlab toolbox DACE , 2002 .

[2]  D. Ginsbourger,et al.  Kriging is well-suited to parallelize optimization , 2010 .

[3]  Andreas Christmann,et al.  Support vector machines , 2008, Data Mining and Knowledge Discovery Handbook.

[4]  Jonas Mockus,et al.  On Bayesian Methods for Seeking the Extremum , 1974, Optimization Techniques.

[5]  Mark D. Hill,et al.  Amdahl's Law in the Multicore Era , 2008, Computer.

[6]  D. Montgomery,et al.  Comparing Computer Experiments for the Gaussian Process Model Using Integrated Prediction Variance , 2013 .

[7]  Yan Wang,et al.  An Efficient First-Principles Saddle Point Searching Method Based on Distributed Kriging Metamodels , 2018 .

[8]  Yves Deville,et al.  DiceKriging, DiceOptim: Two R Packages for the Analysis of Computer Experiments by Kriging-Based Metamodeling and Optimization , 2012 .

[9]  Donald R. Jones,et al.  Global versus local search in constrained optimization of computer models , 1998 .

[10]  Donald R. Jones,et al.  Efficient Global Optimization of Expensive Black-Box Functions , 1998, J. Glob. Optim..

[11]  Luc Pronzato,et al.  Spectral Approximation of the IMSE Criterion for Optimal Designs in Kernel-Based Interpolation Models , 2014, SIAM/ASA J. Uncertain. Quantification.

[12]  Barry L. Nelson,et al.  Stochastic kriging for simulation metamodeling , 2008, 2008 Winter Simulation Conference.

[13]  Peter I. Frazier,et al.  The Parallel Knowledge Gradient Method for Batch Bayesian Optimization , 2016, NIPS.

[14]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[15]  Trevor Hastie,et al.  Multi-class AdaBoost ∗ , 2009 .

[16]  T. Hughes,et al.  A theoretical framework for Petrov-Galerkin methods with discontinuous weighting functions: application to the streamline-upwind procedure. , 1982 .

[17]  Philipp Hennig,et al.  Entropy Search for Information-Efficient Global Optimization , 2011, J. Mach. Learn. Res..

[18]  Jon Louis Bentley,et al.  Multidimensional binary search trees used for associative searching , 1975, CACM.

[19]  Matthew W. Hoffman,et al.  Predictive Entropy Search for Efficient Global Optimization of Black-box Functions , 2014, NIPS.

[20]  Yan Wang,et al.  A robust optimization approach based on multi-fidelity metamodel , 2018 .

[21]  Petros Koumoutsakos,et al.  Reducing the Time Complexity of the Derandomized Evolution Strategy with Covariance Matrix Adaptation (CMA-ES) , 2003, Evolutionary Computation.

[22]  Zi Wang,et al.  Batched High-dimensional Bayesian Optimization via Structural Kernel Learning , 2017, ICML.

[23]  Adam D. Bull,et al.  Convergence Rates of Efficient Global Optimization Algorithms , 2011, J. Mach. Learn. Res..

[24]  P. Spalart.  A One-Equation Turbulence Model for Aerodynamic Flows , 1992 .

[25]  Zoubin Ghahramani,et al.  Parallel Predictive Entropy Search for Batch Global Optimization of Expensive Objective Functions , 2015, NIPS.

[26]  Cheng Li,et al.  Budgeted Batch Bayesian Optimization , 2016, 2016 IEEE 16th International Conference on Data Mining (ICDM).

[27]  Bernhard Sendhoff,et al.  Robust Optimization - A Comprehensive Survey , 2007 .

[28]  D. Ginsbourger,et al.  A Multi-points Criterion for Deterministic Parallel Global Optimization based on Gaussian Processes , 2008 .

[29]  Pushmeet Kohli,et al.  Batched Gaussian Process Bandit Optimization via Determinantal Point Processes , 2016, NIPS.

[30]  Olaf Schenk,et al.  Solving unsymmetric sparse systems of linear equations with PARDISO , 2002, Future Gener. Comput. Syst..

[31]  Nicolas Vayatis,et al.  Parallel Gaussian Process Optimization with Upper Confidence Bound and Pure Exploration , 2013, ECML/PKDD.

[32]  J. Mockus,et al.  The Bayesian approach to global optimization , 1989 .

[33]  David Ginsbourger,et al.  Fast Computation of the Multi-Points Expected Improvement with Applications in Batch Selection , 2013, LION.

[34]  Krishnan V. Pagalthivarthi,et al.  Determination of wear coefficients for erosive wear prediction through Coriolis wear testing , 2005 .

[35]  Nando de Freitas,et al.  Taking the Human Out of the Loop: A Review of Bayesian Optimization , 2016, Proceedings of the IEEE.

[36]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.

[37]  Constantine Caramanis,et al.  Theory and Applications of Robust Optimization , 2010, SIAM Rev..

[38]  Jasper Snoek,et al.  Practical Bayesian Optimization of Machine Learning Algorithms , 2012, NIPS.

[39]  Søren Nymand Lophaven,et al.  DACE - A Matlab Kriging Toolbox , 2002 .

[40]  Daniel Hern'andez-Lobato,et al.  Predictive Entropy Search for Multi-objective Bayesian Optimization with Constraints , 2016, Neurocomputing.

[41]  R. Visintainer,et al.  Finite Element Prediction of Multi-Size Particulate Flow through Three-Dimensional Channel: Code Validation , 2013 .

[43]  David Ginsbourger,et al.  Efficient batch-sequential Bayesian optimization with moments of truncated Gaussian vectors , 2016, 1609.02700.

[44]  Kian Hsiang Low,et al.  Distributed Batch Gaussian Process Optimization , 2017, ICML.

[45]  David Ginsbourger,et al.  Differentiating the Multipoint Expected Improvement for Optimal Batch Design , 2015, MOD.

[46]  A. Basudhar,et al.  Constrained efficient global optimization with support vector machines , 2012, Structural and Multidisciplinary Optimization.

[47]  Victor Picheny,et al.  Bayesian optimization under mixed constraints with a slack-variable augmented Lagrangian , 2016, NIPS.

[48]  Matthew W. Hoffman,et al.  Predictive Entropy Search for Bayesian Optimization with Unknown Constraints , 2015, ICML.

[49]  Neil D. Lawrence,et al.  Batch Bayesian Optimization via Local Penalization , 2015, AISTATS.

[50]  James M. Parr,et al.  Infill sampling criteria for surrogate-based optimization with constraint handling , 2012 .

[51]  Luc Pronzato,et al.  Convex relaxation for IMSE optimal design in random-field models , 2017, Comput. Stat. Data Anal..

[52]  Bolei Zhou,et al.  Optimization as Estimation with Gaussian Processes in Bandit Settings , 2015, AISTATS.

[53]  Alan Fern,et al.  Batch Bayesian Optimization via Simulation Matching , 2010, NIPS.

[54]  Matt J. Kusner,et al.  Bayesian Optimization with Inequality Constraints , 2014, ICML.

[55]  Régis Duvigneau,et al.  A classification approach to efficient global optimization in presence of non-computable domains , 2018 .

[56]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[57]  Harold J. Kushner,et al.  A New Method of Locating the Maximum Point of an Arbitrary Multipeak Curve in the Presence of Noise , 1964 .

[58]  Johan A. K. Suykens,et al.  Least Squares Support Vector Machine Classifiers , 1999, Neural Processing Letters.

[59]  Nando de Freitas,et al.  A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning , 2010, ArXiv.

[60]  Andreas Krause,et al.  Parallelizing Exploration-Exploitation Tradeoffs with Gaussian Process Bandit Optimization , 2012, ICML.