Benchmarking Feature-based Algorithm Selection Systems for Black-box Numerical Optimization

Feature-based algorithm selection aims to automatically find the best one from a portfolio of optimization algorithms on an unseen problem based on its landscape features. Feature-based algorithm selection has recently received attention in the research field of black-box numerical optimization. However, algorithm selection for black-box optimization has been poorly understood. Most previous studies have focused only on whether an algorithm selection system can outperform the single-best solver in a portfolio. In addition, a benchmarking methodology for algorithm selection systems has not been well investigated in the literature. In this context, this paper analyzes algorithm selection systems on the 24 noiseless black-box optimization benchmarking functions. First, we demonstrate that the successful performance 1 measure is more reliable than the expected runtime measure for benchmarking algorithm selection systems. Then, we examine the influence of randomness on the performance of algorithm selection systems. We also show that the performance of algorithm selection systems can be significantly improved by using a pre-solver. We point out that the difficulty of outperforming the single-best solver depends on algorithm portfolios, cross-validation methods, and dimensions. Finally, we demonstrate that the effectiveness of algorithm portfolios depends on various factors. These findings provide fundamental insights for algorithm selection for black-box optimization.

[1]  N. Hansen.  A global surrogate assisted CMA-ES , 2019, GECCO.

[2]  Bernd Bischl,et al.  Exploratory landscape analysis , 2011, GECCO '11.

[3]  Kevin Leyton-Brown,et al.  An evaluation of sequential model-based optimization for expensive blackbox functions , 2013, GECCO.

[4]  Lars Kotthoff,et al.  Algorithm Selection for Combinatorial Search Problems: A Survey , 2012, AI Mag..

[5]  Asma Atamna,et al.  Benchmarking IPOP-CMA-ES-TPA and IPOP-CMA-ES-MSR on the BBOB Noiseless Testbed , 2015, GECCO.

[6]  Nikolaus Hansen,et al.  Invariance, Self-Adaptation and Correlated Mutations and Evolution Strategies , 2000, PPSN.

[7]  Barry O'Sullivan,et al.  Statistical Regimes and Runtime Prediction , 2015, IJCAI.

[8]  László Pál,et al.  Benchmarking a hybrid multi level single linkagealgorithm on the bbob noiseless testbed , 2013, GECCO.

[9]  Benjamin Doerr,et al.  Exploratory Landscape Analysis is Strongly Sensitive to the Sampling Strategy , 2020, PPSN.

[10]  L. Darrell Whitley,et al.  The dispersion metric and the CMA evolution strategy , 2006, GECCO.

[11]  Mahmoud Fouz,et al.  BBOB: Nelder-Mead with resize and halfruns , 2009, GECCO '09.

[12]  Hao Wang,et al.  Neural Network Design: Learning from Neural Architecture Search , 2020, 2020 IEEE Symposium Series on Computational Intelligence (SSCI).

[13]  László Pál,et al.  Comparison of multistart global optimization algorithms on the BBOB noiseless testbed , 2013, GECCO.

[14]  Heike Trautmann,et al.  Detecting Funnel Structures by Means of Exploratory Landscape Analysis , 2015, GECCO.

[15]  Bilel Derbel,et al.  Algorithm selection of anytime algorithms , 2020, GECCO.

[16]  Martin Holena,et al.  Comparison of ordinal and metric gaussian process regression as surrogate models for CMA evolution strategy , 2017, GECCO.

[17]  Anne Auger,et al.  Comparing results of 31 algorithms from the black-box optimization benchmarking BBOB-2009 , 2010, GECCO '10.

[18]  Bilel Derbel,et al.  Multiobjectivization with NSGA-ii on the noiseless BBOB testbed , 2013, GECCO.

[19]  Sébastien Vérel,et al.  Landscape features and automated algorithm selection for multi-objective interpolated continuous optimisation problems , 2021, GECCO.

[20]  Kevin Leyton-Brown,et al.  SATzilla: Portfolio-based Algorithm Selection for SAT , 2008, J. Artif. Intell. Res..

[21]  Jakub Repický,et al.  Gaussian Process Surrogate Models for the CMA Evolution Strategy , 2019, Evolutionary Computation.

[22]  Raymond Ros,et al.  Benchmarking the BFGS algorithm on the BBOB-2009 function testbed , 2009, GECCO '09.

[23]  Heike Trautmann,et al.  Automated Algorithm Selection on Continuous Black-Box Problems by Combining Exploratory Landscape Analysis and Machine Learning , 2017, Evolutionary Computation.

[24]  Mario A. Muñoz,et al.  ICARUS: Identification of complementary algorithms by uncovered sets , 2016, 2016 IEEE Congress on Evolutionary Computation (CEC).

[25]  Anne Auger,et al.  Benchmarking the local metamodel CMA-ES on the noiseless BBOB'2013 test bed , 2013, GECCO.

[26]  Lars Kotthoff,et al.  LLAMA: Leveraging Learning to Automatically Manage Algorithms , 2013, ArXiv.

[27]  Michael Kirley,et al.  Sampling Effects on Algorithm Selection for Continuous Black-Box Optimization , 2021, Algorithms.

[28]  Ramana V. Grandhi,et al.  Improved Distributed Hypercube Sampling , 2002 .

[29]  Sébastien Vérel,et al.  New features for continuous exploratory landscape analysis based on the SOO tree , 2019, FOGA '19.

[30]  Heike Trautmann,et al.  Leveraging TSP Solver Complementarity through Machine Learning , 2018, Evolutionary Computation.

[31]  Rainer Storn,et al.  Differential Evolution – A Simple and Efficient Heuristic for global Optimization over Continuous Spaces , 1997, J. Glob. Optim..

[32]  R. Geoff Dromey,et al.  An algorithm for the selection problem , 1986, Softw. Pract. Exp..

[33]  Thomas Stützle,et al.  Automatically improving the anytime behaviour of optimisation algorithms , 2014, Eur. J. Oper. Res..

[34]  Kevin Leyton-Brown,et al.  Sequential Model-Based Optimization for General Algorithm Configuration , 2011, LION.

[35]  Bernd Bischl,et al.  Cell Mapping Techniques for Exploratory Landscape Analysis , 2014 .

[36]  F. Hutter,et al.  Hydra-MIP : Automated Algorithm Configuration and Selection for Mixed Integer Programming , 2011 .

[37]  Olivier Teytaud,et al.  Versatile black-box optimization , 2020, GECCO.

[38]  Yang Lou,et al.  Exploratory landscape analysis using algorithm based sampling , 2018, GECCO.

[39]  Nikolaus Hansen,et al.  Benchmarking a BI-population CMA-ES on the BBOB-2009 function testbed , 2009, GECCO '09.

[40]  Anne Auger,et al.  Comparing the (1+1)-CMA-ES with a mirrored (1+2)-CMA-ES with sequential selection on the noiseless BBOB-2010 testbed , 2010, GECCO '10.

[41]  Bernd Bischl,et al.  ASlib: A benchmark library for algorithm selection , 2015, Artif. Intell..

[42]  Petr Posík,et al.  Dimension Selection in Axis-Parallel Brent-STEP Method for Black-Box Optimization of Separable Continuous Functions , 2015, GECCO.

[43]  Kate Smith-Miles,et al.  Cross-disciplinary perspectives on meta-learning for algorithm selection , 2009, CSUR.

[44]  Yuri Malitsky,et al.  ISAC - Instance-Specific Algorithm Configuration , 2010, ECAI.

[45]  Antonio LaTorre,et al.  Benchmarking a MOS-based algorithm on the BBOB-2010 noiseless function testbed , 2010, GECCO '10.

[46]  Rémi Munos,et al.  Optimistic Optimization of Deterministic Functions , 2011, NIPS 2011.

[47]  Youhei Akimoto,et al.  Benchmarking the PSA-CMA-ES on the BBOB noiseless testbed , 2018, GECCO.

[48]  Eckart Zitzler,et al.  HypE: An Algorithm for Fast Hypervolume-Based Many-Objective Optimization , 2011, Evolutionary Computation.

[49]  Anne Auger,et al.  Experimental Comparisons of Derivative Free Optimization Algorithms , 2009, SEA.

[50]  Xin Yao,et al.  Population-based Algorithm Portfolios with automated constituent algorithms selection , 2014, Inf. Sci..

[51]  Marius Thomas Lindauer,et al.  AutoFolio: An Automatically Configured Algorithm Selector , 2015, J. Artif. Intell. Res..

[52]  Kevin Leyton-Brown,et al.  Bias in Algorithm Portfolio Performance Evaluation , 2016, IJCAI.

[53]  Kevin Leyton-Brown,et al.  Hydra: Automatically Configuring Algorithms for Portfolio-Based Selection , 2010, AAAI.

[54]  Yuri Malitsky,et al.  Algorithm Selection and Scheduling , 2011, CP.

[55]  Anne Auger,et al.  COCO: a platform for comparing continuous optimizers in a black-box setting , 2016, Optim. Methods Softw..

[56]  Kate Smith-Miles,et al.  Generating New Space-Filling Test Instances for Continuous Black-Box Optimization , 2020, Evolutionary Computation.

[57]  Bernd Bischl,et al.  Algorithm selection based on exploratory landscape analysis and cost-sensitive learning , 2012, GECCO '12.

[58]  Heike Trautmann,et al.  Automated Algorithm Selection: Survey and Perspectives , 2018, Evolutionary Computation.

[59]  Petr Posík,et al.  BBOB-benchmarking two variants of the line-search algorithm , 2009, GECCO '09.

[60]  Carola Doerr,et al.  Landscape-aware fixed-budget performance regression and algorithm selection for modular CMA-ES variants , 2020, GECCO.

[61]  Petr Posík,et al.  Online Black-Box Algorithm Portfolios for Continuous Optimization , 2014, PPSN.

[62]  Anne Auger,et al.  COCO: Performance Assessment , 2016, ArXiv.

[63]  Ofer M. Shir,et al.  Benchmarking discrete optimization heuristics with IOHprofiler , 2019, GECCO.

[64]  Anne Auger,et al.  Performance evaluation of an advanced local search evolutionary algorithm , 2005, 2005 IEEE Congress on Evolutionary Computation.

[65]  Yuri Malitsky,et al.  Features for Exploiting Black-Box Optimization Problem Structure , 2013, LION.

[66]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[67]  Francisco Herrera,et al.  A practical tutorial on the use of nonparametric statistical tests as a methodology for comparing evolutionary and swarm intelligence algorithms , 2011, Swarm Evol. Comput..

[68]  Michèle Sebag,et al.  Bi-population CMA-ES agorithms with surrogate models and line searches , 2013, GECCO.

[69]  Bilel Derbel,et al.  Experiments on Greedy and Local Search Heuristics for ddimensional Hypervolume Subset Selection , 2016, GECCO.

[70]  Anne Auger,et al.  Real-Parameter Black-Box Optimization Benchmarking 2009: Noiseless Functions Definitions , 2009 .

[71]  Hao Wang,et al.  Algorithm configuration data mining for CMA evolution strategies , 2017, GECCO.

[72]  Tome Eftimov,et al.  Towards Feature-Based Performance Regression Using Trajectory Data , 2021, EvoApplications.

[73]  Tome Eftimov,et al.  The impact of hyper-parameter tuning for landscape-aware performance regression and algorithm selection , 2021, GECCO.

[74]  Pascal Kerschke,et al.  Comprehensive Feature-Based Landscape Analysis of Continuous and Constrained Optimization Problems Using the R-Package Flacco , 2017, Studies in Classification, Data Analysis, and Knowledge Organization.

[75]  Duc Manh Nguyen.  Benchmarking a variant of the CMAES-APOP on the BBOB noiseless testbed , 2018, GECCO.

[76]  The Hessian Estimation Evolution Strategy , 2020, PPSN.

[77]  Saman K. Halgamuge,et al.  Exploratory Landscape Analysis of Continuous Space Optimization Problems Using Information Content , 2015, IEEE Transactions on Evolutionary Computation.

[78]  Greg Hamerly,et al.  Learning the k in k-means , 2003, NIPS.