EXPObench: Benchmarking Surrogate-based Optimisation Algorithms on Expensive Black-box Functions

Surrogate algorithms such as Bayesian optimisation are especially designed for black-box optimisation problems with expensive objectives, such as hyperparameter tuning or simulation-based optimisation. In the literature, these algorithms are usually evaluated with synthetic benchmarks which are well established but have no expensive objective, and only on one or two real-life applications which vary wildly between papers. There is a clear lack of standardisation when it comes to benchmarking surrogate algorithms on real-life, expensive, black-box objective functions. This makes it very difficult to draw conclusions on the effect of algorithmic contributions. A new benchmark library, EXPObench, provides first steps towards such a standardisation. The library is used to provide an extensive comparison of six different surrogate algorithms on four expensive optimisation problems from different reallife applications. This has led to new insights regarding the relative importance of exploration, the evaluation time of the objective, and the used model. A further contribution is that we make the algorithms and benchmark problem instances publicly available, contributing to more uniform analysis of surrogate algorithms. Most importantly, we include the performance of the six algorithms on all evaluated problem instances. This results in a unique new dataset that lowers the bar for researching new methods as the number of expensive evaluations required for comparison is significantly reduced.

[1]  Marius Lindauer,et al.  DACBench: A Benchmark Library for Dynamic Algorithm Configuration , 2021, IJCAI.

[2]  Jonas Mockus,et al.  On Bayesian Methods for Seeking the Extremum , 1974, Optimization Techniques.

[3]  Thomas Bartz-Beielstein,et al.  Model-based methods for continuous and discrete global optimization , 2017, Appl. Soft Comput..

[4]  Kevin Leyton-Brown,et al.  Sequential Model-Based Optimization for General Algorithm Configuration , 2011, LION.

[5]  Pascal Kerschke,et al.  Single- and multi-objective game-benchmark for evolutionary algorithms , 2019, GECCO.

[6]  Sebastian Thrun,et al.  Learning to Learn: Introduction and Overview , 1998, Learning to Learn.

[7]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[8]  Mathijs de Weerdt,et al.  Black-box mixed-variable optimisation using a surrogate model that satisfies integer constraints , 2020, GECCO Companion.

[9]  Aaron Klein,et al.  BOHB: Robust and Efficient Hyperparameter Optimization at Scale , 2018, ICML.

[10]  Nando de Freitas,et al.  Taking the Human Out of the Loop: A Review of Bayesian Optimization , 2016, Proceedings of the IEEE.

[11]  Javier Marenco,et al.  Machine learning for improving performance in an evolutionary algorithm for minimum path with uncertain costs given by massively simulated scenarios , 2021, ArXiv.

[12]  Margret Keuper,et al.  NAS-Bench-301 and the Case for Surrogate Benchmarks for Neural Architecture Search , 2020, ArXiv.

[13]  Michel Verhaegen,et al.  Online Optimization With Costly and Noisy Measurements Using Random Fourier Expansions , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[14]  Identifying Properties of Real-World Optimisation Problems through a Questionnaire , 2020, ArXiv.

[15]  Stefan Görtz,et al.  Improving variable-fidelity surrogate modeling via gradient-enhanced kriging and a generalized hybrid bridge function , 2013 .

[16]  Sebastian Curi,et al.  Safe Contextual Bayesian Optimization for Sustainable Room Temperature PID Control Tuning , 2019, IJCAI.

[17]  Pramudita Satria Palar,et al.  On the use of surrogate models in engineering design optimization and exploration: the key issues , 2019, GECCO.

[18]  Yi Yang,et al.  NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search , 2020, ICLR.

[19]  Johann Dréo,et al.  Making a case for (Hyper-)parameter tuning as benchmark problems , 2019, GECCO.

[20]  Sanja Petrovic,et al.  HyFlex: A Benchmark Framework for Cross-Domain Heuristic Search , 2011, EvoCOP.

[21]  Mathijs de Weerdt,et al.  Black-box Combinatorial Optimization using Models with Integer-valued Minima , 2019, ArXiv.

[22]  Alberto Costa,et al.  RBFOpt: an open-source library for black-box optimization with costly function evaluations , 2018, Mathematical Programming Computation.

[23]  Giancarlo Mauri,et al.  Biochemical parameter estimation vs. benchmark functions: A comparative study of optimization performance and representation design , 2019, Appl. Soft Comput..

[24]  Zhonghua Han,et al.  COMPARISON OF INFILL SAMPLING CRITERIA IN KRIGING-BASED AERODYNAMIC OPTIMIZATION , 2012 .

[25]  Fabio Caraffini.  The SOS Platform: Designing, Tuning and Statistically Benchmarking Optimisation Algorithms , 2020, Mathematics.

[26]  Kate Smith-Miles,et al.  Towards objective measures of algorithm performance across instance space , 2014, Comput. Oper. Res..

[27]  Aaron Klein,et al.  NAS-Bench-101: Towards Reproducible Neural Architecture Search , 2019, ICML.

[28]  Bernd Bischl,et al.  An Open Source AutoML Benchmark , 2019, ArXiv.

[29]  Michael A. Osborne,et al.  Bayesian Optimisation over Multiple Continuous and Categorical Inputs , 2019, ICML.

[30]  Sébastien Vérel,et al.  ParadisEO-MO: from fitness landscape analysis to efficient local search algorithms , 2013, Journal of Heuristics.

[31]  Piet Demeester,et al.  A Surrogate Modeling and Adaptive Sampling Toolbox for Computer Based Design , 2010, J. Mach. Learn. Res..

[32]  Michael Affenzeller,et al.  HeuristicLab: A Generic and Extensible Optimization Environment , 2005 .

[33]  Yoshua Bengio,et al.  Random Search for Hyper-Parameter Optimization , 2012, J. Mach. Learn. Res..

[34]  Johannes M. Dieterich,et al.  Empirical review of standard benchmark functions using evolutionary global optimization , 2012, ArXiv.

[35]  David D. Cox,et al.  Making a Science of Model Search: Hyperparameter Optimization in Hundreds of Dimensions for Vision Architectures , 2013, ICML.

[36]  El Houssaine Aghezzaf,et al.  Solving the stochastic time-dependent orienteering problem with time windows , 2016, Eur. J. Oper. Res..

[37]  Mathijs de Weerdt,et al.  Continuous surrogate-based optimization algorithms are well-suited for expensive discrete problems , 2020, BNAIC/BENELEARN.

[38]  Vanessa Volz,et al.  On benchmarking surrogate-assisted evolutionary algorithms , 2019, GECCO.

[39]  Tony Pourmohamad.  CompModels: A Suite of Computer Model Test Functions for Bayesian Optimization , 2020, R J..

[40]  Jasper Snoek,et al.  Practical Bayesian Optimization of Machine Learning Algorithms , 2012, NIPS.

[41]  Joaquim R. R. A. Martins,et al.  A Python surrogate modeling framework with derivatives , 2019, Adv. Eng. Softw..

[42]  Tonio Buonassisi,et al.  Benchmarking the performance of Bayesian optimization across multiple experimental materials science domains , 2021, npj Computational Materials.

[43]  Masataka Goto,et al.  Generative Melody Composition with Human-in-the-Loop Bayesian Optimization , 2020, ArXiv.

[44]  Thomas Bartz-Beielstein,et al.  Comparison of parallel surrogate-assisted optimization approaches , 2018, GECCO.

[45]  Christine A. Shoemaker,et al.  pySOT and POAP: An event-driven asynchronous framework for surrogate optimization , 2019, ArXiv.

[46]  Tianqi Chen,et al.  XGBoost: A Scalable Tree Boosting System , 2016, KDD.

[47]  Hao Wang,et al.  IOHprofiler: A Benchmarking and Profiling Tool for Iterative Optimization Heuristics , 2018, ArXiv.

[48]  Jonathan E. Fieldsend,et al.  A Suite of Computationally Expensive Shape Optimisation Problems Using Computational Fluid Dynamics , 2018, PPSN.