Benchmarking discrete optimization heuristics with IOHprofiler

Automated benchmarking environments aim to support researchers in understanding how different algorithms perform on different types of optimization problems. Such comparisons carry the potential to provide insights into the strengths and weaknesses of different approaches, which can be leveraged into designing new algorithms. Carefully selected benchmark problems are also needed as training sets in the context of algorithm selection and configuration. With the ultimate goal to create a meaningful benchmark set for iterative optimization heuristics, we compile and assess in this work a selection of discrete optimization problems that subscribe to different types of fitness landscapes. All problems have been implemented and tested within IOHprofiler, our recently released software built to assess iterative heuristics solving combinatorial optimization problems. For each selected problem we compare performances of eleven different heuristics. Apart from fixed-target and fixed-budget results for the individual problems, we also derive ECDF results for groups of problems. To obtain these, we have implemented an add-on for IOHprofiler which allows aggregation of performance data across different benchmark functions.

[1]  Marco Zaffalon,et al.  Time for a change: a tutorial for comparing multiple classifiers through Bayesian analysis , 2016, J. Mach. Learn. Res..

[2]  Benjamin Doerr,et al.  From black-box complexity to designing new genetic algorithms , 2015, Theor. Comput. Sci..

[3]  Benjamin Doerr,et al.  Black-Box Complexity: Breaking the O(n logn) Barrier of LeadingOnes , 2011, Artificial Evolution.

[4]  Carola Doerr,et al.  Hyper-parameter tuning for the (1 + (λ, λ)) GA , 2019, GECCO.

[5]  Robert Tibshirani,et al.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd Edition , 2001, Springer Series in Statistics.

[6]  Carola Doerr,et al.  Complexity Theory for Discrete Black-Box Optimization Heuristics , 2018, Theory of Evolutionary Computation.

[7]  Thomas Bäck,et al.  Interpolating Local and Global Search by Controlling the Variance of Standard Bit Mutation , 2019, 2019 IEEE Congress on Evolutionary Computation (CEC).

[8]  Brett Stevens,et al.  A survey of known results and research areas for n-queens , 2009, Discret. Math..

[9]  Anne Auger,et al.  Comparing results of 31 algorithms from the black-box optimization benchmarking BBOB-2009 , 2010, GECCO '10.

[10]  H. Mühlenbein,et al.  From Recombination of Genes to the Estimation of Distributions I. Binary Parameters , 1996, PPSN.

[11]  Thomas Jansen,et al.  On the analysis of the (1+1) evolutionary algorithm , 2002, Theor. Comput. Sci..

[12]  Stefan M. Wild,et al.  Benchmarking Derivative-Free Optimization Algorithms , 2009, SIAM J. Optim..

[13]  Benjamin Doerr,et al.  Optimal Parameter Choices via Precise Black-Box Analysis , 2016, GECCO.

[14]  Sébastien Vérel,et al.  Multifractality and dimensional determinism in local optima networks , 2018, GECCO.

[15]  David B. Fogel,et al.  Evolutionary algorithms in theory and practice , 1997, Complex.

[16]  Stephan Mertens,et al.  Low autocorrelation binary sequences , 2015, 1512.02475.

[17]  Ofer M. Shir,et al.  Benchmarking discrete optimization heuristics with IOHprofiler , 2020, Appl. Soft Comput..

[18]  J. Franklin,et al.  The elements of statistical learning: data mining, inference and prediction , 2005 .

[19]  Hao Wang,et al.  Towards a theory-guided benchmarking suite for discrete black-box optimization heuristics: profiling (1 + λ) EA variants on onemax and leadingones , 2018, GECCO.

[20]  Anne Auger,et al.  COCO: Performance Assessment , 2016, ArXiv.

[21]  Antonio J. Nebro,et al.  jMetal: A Java framework for multi-objective optimization , 2011, Adv. Eng. Softw..

[22]  Carsten Witt,et al.  Theory of estimation-of-distribution algorithms , 2018, GECCO.

[23]  Thomas Weise,et al.  Difficult features of combinatorial optimization problems and the tunable w-model benchmark problem for simulating them , 2018, GECCO.

[24]  Paul Erdgs,et al.  ON TWO PROBLEMS OF INFORMATION THEORY bY PAUL ERDGS and ALFRJ~D RgNYI , 2001 .

[25]  I. A. Pasha,et al.  Bi-alphabetic pulse compression radar signal design , 2000 .

[26]  Hao Wang,et al.  IOHprofiler: A Benchmarking and Profiling Tool for Iterative Optimization Heuristics , 2018, ArXiv.

[27]  Bernd Bischl,et al.  Exploratory landscape analysis , 2011, GECCO '11.

[28]  Alessandro Panconesi,et al.  Concentration of Measure for the Analysis of Randomized Algorithms , 2009 .

[29]  Andrew Lucas,et al.  Ising formulations of many NP problems , 2013, Front. Physics.

[30]  Arina Buzdalova,et al.  Offspring population size matters when comparing evolutionary algorithms with self-adjusting mutation rates , 2019, GECCO.

[31]  Benjamin Doerr,et al.  The (1+λ) evolutionary algorithm with self-adjusting mutation rate , 2017, GECCO.

[32]  Kurt Mehlhorn,et al.  The Query Complexity of a Permutation-Based Variant of Mastermind , 2019, Discret. Appl. Math..

[33]  Carlos M. Fonseca,et al.  On the Computation of the Empirical Attainment Function , 2011, EMO.

[34]  Goldberg,et al.  Genetic algorithms , 1993, Robust Control Systems with Genetic Algorithms.

[35]  Carsten Witt,et al.  Tight Bounds on the Optimization Time of a Randomized Search Heuristic on Linear Functions† , 2013, Combinatorics, Probability and Computing.

[36]  Ofer M. Shir,et al.  Bayesian performance analysis for black-box optimization benchmarking , 2019, GECCO.

[37]  Benjamin Doerr,et al.  Unknown solution length problems with no asymptotically optimal run time , 2017, GECCO.

[38]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[39]  Thomas Bäck,et al.  Intelligent Mutation Rate Control in Canonical Genetic Algorithms , 1996, ISMIS.

[40]  Anne Auger,et al.  COCO: The Bi-objective Black Box Optimization Benchmarking (bbob-biobj) Test Suite , 2016, ArXiv.

[41]  Anne Auger,et al.  COCO: a platform for comparing continuous optimizers in a black-box setting , 2016, Optim. Methods Softw..

[42]  Benjamin Doerr,et al.  The ($$1+\lambda $$1+λ) Evolutionary Algorithm with Self-Adjusting Mutation Rate , 2018, Algorithmica.

[43]  Dirk Sudholt,et al.  Crossover is provably essential for the Ising model on trees , 2005, GECCO '05.

[44]  Benjamin Doerr,et al.  Multiplicative Drift Analysis , 2010, GECCO '10.

[45]  Sébastien Vérel,et al.  On the Fractal Nature of Local Optima Networks , 2018, EvoCOP.

[46]  José Antonio Lozano,et al.  Bayesian inference for algorithm ranking analysis , 2018, GECCO.

[47]  Marc Schoenauer,et al.  Per instance algorithm configuration of CMA-ES with limited budget , 2017, GECCO.

[48]  Dimo Brockhoff,et al.  Mixed-integer benchmark problems for single- and bi-objective optimization , 2019, GECCO.

[49]  Per Kristian Lehre,et al.  Black-Box Search by Unbiased Variation , 2010, GECCO '10.

[50]  Heike Trautmann,et al.  The R-Package FLACCO for exploratory landscape analysis with applications to multi-objective optimization problems , 2016, 2016 IEEE Congress on Evolutionary Computation (CEC).

[51]  Ingo Wegener,et al.  The one-dimensional Ising model: Mutation versus recombination , 2005, Theor. Comput. Sci..

[52]  Dieter Beule,et al.  Evolutionary search for low autocorrelated binary sequences , 1998, IEEE Trans. Evol. Comput..

[53]  Ofer M. Shir,et al.  Predict or screen your expensive assay: DoE vs. surrogates in experimental combinatorial optimization , 2019, GECCO.

[54]  R. P. Ingalls,et al.  FOURTH TEST OF GENERAL RELATIVITY: PRELIMINARY RESULTS. , 1968 .

[55]  F. Barahona.  On the computational complexity of Ising spin glass models , 1982 .

[56]  Nikolaus Hansen,et al.  A practical guide to experimentation , 2018, GECCO.

[57]  Vincent A. Mellor,et al.  Numerical Simulations of the Ising Model on the Union Jack Lattice , 2010, 1106.5579.

[58]  Angelika Steger,et al.  The linear hidden subset problem for the (1 + 1) EA with scheduled and adaptive mutation rates , 2018, GECCO.

[59]  Ofer M. Shir,et al.  Compiling a benchmarking test-suite for combinatorial black-box optimization: a position paper , 2018, GECCO.

[60]  Carola Doerr,et al.  The (1+1) Elitist Black-Box Complexity of LeadingOnes , 2016, GECCO.

[61]  Benjamin Doerr,et al.  Analyzing randomized search heuristics via stochastic domination , 2019, Theor. Comput. Sci..

[62]  Szymon Wasik,et al.  Optil.io: Cloud Based Platform For Solving Optimization Problems Using Crowdsourcing Approach , 2016, CSCW '16 Companion.

[63]  Benjamin Doerr,et al.  Optimal Static and Self-Adjusting Parameter Choices for the (1+(λ,λ))\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$( , 2017, Algorithmica.

[64]  Dirk Thierens.  On benchmark properties for adaptive operator selection , 2009, GECCO '09.

[65]  S. Kauffman,et al.  Towards a general theory of adaptive walks on rugged landscapes. , 1987, Journal of theoretical biology.

[66]  Thomas Bäck,et al.  An evolutionary heuristic for the maximum independent set problem , 1994, Proceedings of the First IEEE Conference on Evolutionary Computation. IEEE World Congress on Computational Intelligence.

[67]  Heike Trautmann,et al.  Automated Algorithm Selection: Survey and Perspectives , 2018, Evolutionary Computation.

[68]  Benjamin Doerr,et al.  Fast genetic algorithms , 2017, GECCO.

[69]  Heinz Mühlenbein,et al.  The Equation for Response to Selection and Its Use for Prediction , 1997, Evolutionary Computation.

[70]  Thomas Jansen,et al.  The Ising Model: Simple Evolutionary Algorithms as Adaptation Schemes , 2004, PPSN.

[71]  Benjamin Doerr,et al.  Better Runtime Guarantees via Stochastic Domination , 2018, EvoCOP.

[72]  El-Ghazali Talbi,et al.  ParadisEO: A Framework for the Reusable Design of Parallel and Distributed Metaheuristics , 2004, J. Heuristics.

[73]  Marc Schoenauer,et al.  Rigorous Hitting Times for Binary Mutations , 1999, Evolutionary Computation.

[74]  Per Kristian Lehre,et al.  Faster black-box algorithms through higher arity operators , 2010, FOGA '11.