Sequential model based optimization of partially defined functions under unknown constraints

This paper presents a sequential model based optimization framework for optimizing a black-box, multi-extremal and expensive objective function, which is also partially defined, that is it is undefined outside the feasible region. Furthermore, the constraints defining the feasible region within the search space are unknown. The approach proposed in this paper, namely SVM-CBO, is organized in two consecutive phases, the first uses a Support Vector Machine classifier to approximate the boundary of the unknown feasible region, the second uses Bayesian Optimization to find a globally optimal solution within the feasible region. In the first phase the next point to evaluate is chosen by dealing with the trade-off between improving the current estimate of the feasible region and discovering possible disconnected feasible sub-regions. In the second phase, the next point to evaluate is selected as the minimizer of the Lower Confidence Bound acquisition function but constrained to the current estimate of the feasible region. The main of the paper is a comparison with a Bayesian Optimization process which uses a fixed penalty value for infeasible function evaluations, under a limited budget (i.e., maximum number of function evaluations). Results are related to five 2D test functions from literature and 80 test functions, with increasing dimensionality and complexity, generated through the Emmental-type GKLS software. SVM-CBO proved to be significantly more effective as well as computationally efficient.

[1]  Sébastien Le Digabel,et al.  A Taxonomy of Constraints in Simulation-Based Optimization , 2015, 1505.07881.

[2]  V. Grishagin,et al.  Multidimensional Constrained Global Optimization in Domains with Computable Boundaries , 2015 .

[3]  L. Rudenko.  Objective functional approximation in a partially defined optimization problem , 1994 .

[4]  Lars Kotthoff,et al.  Automated Machine Learning: Methods, Systems, Challenges , 2019, The Springer Series on Challenges in Machine Learning.

[5]  Yaroslav D. Sergeyev,et al.  Deterministic Global Optimization: An Introduction to the Diagonal Approach , 2017 .

[6]  Céline Helbert,et al.  Gaussian process optimization with failures: classification and convergence proof , 2020, Journal of Global Optimization.

[7]  Alexander J. Smola,et al.  Learning with Kernels: support vector machines, regularization, optimization, and beyond , 2001, Adaptive computation and machine learning series.

[8]  Gianni Di Pillo,et al.  A DIRECT-type approach for derivative-free constrained global optimization , 2016, Comput. Optim. Appl..

[9]  A. Basudhar,et al.  Constrained efficient global optimization with support vector machines , 2012, Structural and Multidisciplinary Optimization.

[10]  P. Frazier.  Bayesian Optimization , 2018, Hyperparameter Optimization in Machine Learning.

[11]  Matthew W. Hoffman,et al.  Predictive Entropy Search for Bayesian Optimization with Unknown Constraints , 2015, ICML.

[12]  Omkar Kulkarni,et al.  Application of Grasshopper Optimization Algorithm for Constrained and Unconstrained Test Functions , 2017 .

[13]  Francesco Archetti,et al.  Sequential model based optimization with black-box constraints: Feasibility determination via machine learning , 2019 .

[14]  Alkis Gotovos,et al.  Safe Exploration for Optimization with Gaussian Processes , 2015, ICML.

[15]  Kok Lay Teo,et al.  An exact penalty function-based differential search algorithm for constrained global optimization , 2015, Soft Computing.

[16]  A. Zilinskas,et al.  Global optimization based on a statistical model and simplicial partitioning , 2002 .

[17]  Hao Huang,et al.  STOCHASTIC OPTIMIZATION FOR FEASIBILITY DETERMINATION: AN APPLICATION TO WATER PUMP OPERATION IN WATER DISTRIBUTION NETWORK , 2018, 2018 Winter Simulation Conference (WSC).

[18]  Guy L. Curry,et al.  On optimizing certain nonlinear convex functions which are partially defined by a simulation process , 1977, Math. Program..

[19]  N. Zheng,et al.  Global Optimization of Stochastic Black-Box Systems via Sequential Kriging Meta-Models , 2006, J. Glob. Optim..

[20]  Joel W. Burdick,et al.  Stagewise Safe Bayesian Optimization with Gaussian Processes , 2018, ICML.

[21]  Chen Lei,et al.  Automated Machine Learning , 2021, Cognitive Intelligence and Robotics.

[22]  Kevin Leyton-Brown,et al.  Auto-WEKA: combined selection and hyperparameter optimization of classification algorithms , 2012, KDD.

[23]  Nando de Freitas,et al.  Taking the Human Out of the Loop: A Review of Bayesian Optimization , 2016, Proceedings of the IEEE.

[24]  Nando de Freitas,et al.  Portfolio Allocation for Bayesian Optimization , 2010, UAI.

[25]  Roman G. Strongin,et al.  Global optimization with non-convex constraints , 2000 .

[26]  F. Archetti,et al.  A probabilistic algorithm for global optimization , 1979 .

[27]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.

[28]  Donald R. Jones,et al.  Efficient Global Optimization of Expensive Black-Box Functions , 1998, J. Glob. Optim..

[29]  Francesco Archetti,et al.  Bayesian optimization of pump operations in water distribution systems , 2018, J. Glob. Optim..

[30]  Matthew W. Hoffman,et al.  A General Framework for Constrained Bayesian Optimization using Information-based Search , 2015, J. Mach. Learn. Res..

[31]  Frank Hutter,et al.  Neural Architecture Search: A Survey , 2018, J. Mach. Learn. Res..

[32]  L. Grippo,et al.  Exact penalty functions in constrained optimization , 1989 .

[33]  A. A. Zhigli︠a︡vskiĭ,et al.  Stochastic Global Optimization , 2007 .

[34]  V. Donskoi.  Partially defined optimization problems: An approach to a solution that is based on pattern recognition theory , 1993 .

[35]  Yaroslav D. Sergeyev,et al.  Deterministic Global Optimization , 2017 .

[36]  Guilherme Ottoni,et al.  Constrained Bayesian Optimization with Noisy Experiments , 2017, Bayesian Analysis.

[37]  Y. D. Sergeyev,et al.  Global Optimization with Non-Convex Constraints - Sequential and Parallel Algorithms (Nonconvex Optimization and its Applications Volume 45) (Nonconvex Optimization and Its Applications) , 2000 .

[38]  Ya D Sergeyev,et al.  On the efficiency of nature-inspired metaheuristics in expensive global optimization with limited budget , 2018, Scientific Reports.

[39]  Matthias Poloczek,et al.  Bayesian Optimization with Gradients , 2017, NIPS.

[40]  Aaron Klein,et al.  Efficient and Robust Automated Machine Learning , 2015, NIPS.

[41]  D. Ginsbourger,et al.  A benchmark of kriging-based infill criteria for noisy optimization , 2013, Structural and Multidisciplinary Optimization.

[42]  Robert B. Gramacy,et al.  Optimization Under Unknown Constraints , 2010, 1004.4027.

[43]  Régis Duvigneau,et al.  A classification approach to efficient global optimization in presence of non-computable domains , 2018 .

[44]  Julien Bect,et al.  A Bayesian approach to constrained single- and multi-objective optimization , 2015, Journal of Global Optimization.

[45]  Marius Lindauer,et al.  Pitfalls and Best Practices in Algorithm Configuration , 2017, J. Artif. Intell. Res..

[46]  Nando de Freitas,et al.  A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning , 2010, ArXiv.

[47]  Francesco Archetti,et al.  Global optimization in machine learning: the design of a predictive analytics application , 2018, Soft Computing.

[48]  Andreas Krause,et al.  Information-Theoretic Regret Bounds for Gaussian Process Optimization in the Bandit Setting , 2009, IEEE Transactions on Information Theory.

[49]  Gianni Di Pillo,et al.  A Derivative-Free Algorithm for Constrained Global Optimization Based on Exact Penalty Functions , 2013, Journal of Optimization Theory and Applications.

[50]  Yaroslav D. Sergeyev,et al.  Index information algorithm with local tuning for solving multidimensional global optimization problems with multiextremal constraints , 2011, Math. Program..

[51]  Victor Picheny,et al.  Bayesian optimization under mixed constraints with a slack-variable augmented Lagrangian , 2016, NIPS.

[52]  Francesco Archetti,et al.  Bayesian Optimization for Full Waveform Inversion , 2018 .

[53]  Roman G. Strongin,et al.  Introduction to Global Optimization Exploiting Space-Filling Curves , 2013 .

[54]  Yaroslav D. Sergeyev,et al.  Emmental-Type GKLS-Based Multiextremal Smooth Test Problems with Non-linear Constraints , 2017, LION.

[55]  Amir Hajian,et al.  Constrained Bayesian Optimization for Problems with Piece-wise Smooth Constraints , 2018, Canadian Conference on AI.

[56]  P. A. Simionescu,et al.  New Concepts in Graphic Visualization of Objective Functions , 2002, Volume 2: 28th Design Automation Conference.

[57]  A. Zhigljavsky.  Stochastic Global Optimization , 2008, International Encyclopedia of Statistical Science.

[58]  Andreas Christmann,et al.  Support vector machines , 2008, Data Mining and Knowledge Discovery Handbook.

[59]  Sudhanshu K. Mishra,et al.  Some New Test Functions for Global Optimization and Performance of Repulsive Particle Swarm Method , 2006 .

[60]  A. ilinskas,et al.  Global optimization based on a statistical model and simplicial partitioning , 2002 .

[61]  Yaroslav D. Sergeyev,et al.  A one-dimensional local tuning algorithm for solving GO problems with partially defined constraints , 2007, Optim. Lett..

[62]  Julius Zilinskas,et al.  Advantages of simplicial partitioning for Lipschitz optimization problems with linear constraints , 2014, Optimization Letters.