Choosing the right algorithm with hints from complexity theory: (hot-off-the-press track at GECCO 2022)

Choosing a suitable algorithm from the myriads of different search heuristics is difficult when faced with a novel optimization problem. In this work, we argue that the purely academic question of what could be the best possible algorithm in a certain broad class of black-box optimizers can give fruitful indications in which direction to search for good established optimization heuristics. We demonstrate this approach on the recently proposed DLB benchmark, for which the only known results are O(n3) runtimes for several classic evolutionary algorithms and an O(n2 log n) runtime for an estimation-of-distribution algorithm. Our finding that the unary unbiased black-box complexity is only O(n2) suggests the Metropolis algorithm as an interesting candidate and we prove that it solves the DLB problem in quadratic time. Since we also prove that better runtimes cannot be obtained in the class of unary unbiased algorithms, we shift our attention to algorithms that use the information of more parents to generate new solutions. An artificial algorithm of this type having an O(n log n) runtime leads to the result that the significance-based compact genetic algorithm (sig-cGA) can solve the DLB problem also in time O(n log n). This paper for the Hot-of-the-Press track at GECCO 2022 summarizes the work Shouda Wang, Weijie Zheng, Benjamin Doerr: Choosing the Right Algorithm With Hints From Complexity Theory. IJCAI 2021: 1697--1703 [11].

[1]  Carsten Witt,et al.  Self-Adjusting Evolutionary Algorithms for Multimodal Optimization , 2020, Algorithmica.

[2]  F. Chicano.  GECCO 2021 , 2021, ACM SIGEVOlution.

[3]  P. S. Oliveto,et al.  Automatic adaptation of hypermutation rates for multimodal optimisation , 2021, FOGA.

[4]  Dirk Sudholt,et al.  Self-adjusting offspring population sizes outperform fixed parameters on the cliff function , 2021, FOGA.

[5]  Duc-Cuong Dang,et al.  Non-elitist evolutionary algorithms excel in fitness landscapes with sparse deceptive regions and dense valleys , 2021, GECCO.

[6]  Duc-Cuong Dang,et al.  Escaping Local Optima with Non-Elitist Evolutionary Algorithms , 2021, AAAI.

[7]  Xin Yao,et al.  When non-elitism meets time-linkage problems , 2021, GECCO.

[8]  Lazy parameter tuning and control: choosing all parameters randomly from a power-law distribution , 2021, GECCO.

[9]  Benjamin Doerr,et al.  Lower Bounds for Non-Elitist Evolutionary Algorithms via Negative Multiplicative Drift , 2020, Evolutionary Computation.

[10]  X. Yao,et al.  Analysis of Evolutionary Algorithms on Fitness Function With Time-Linkage Property , 2020, IEEE Transactions on Evolutionary Computation.

[11]  Benjamin Doerr,et al.  The Univariate Marginal Distribution Algorithm Copes Well with Deception and Epistasis , 2020, Evolutionary Computation.

[12]  Weijie Zheng,et al.  Sharp Bounds for Genetic Drift in Estimation of Distribution Algorithms , 2020, IEEE Transactions on Evolutionary Computation.

[13]  Benjamin Doerr,et al.  A gentle introduction to theory (for non-theoreticians) , 2020, GECCO Companion.

[14]  Benjamin Doerr,et al.  A Tight Runtime Analysis for the (μ+λ)\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${(\mu + \lambda )}$$\end{documen , 2020, Algorithmica.

[15]  Benjamin Doerr,et al.  First Steps Towards a Runtime Analysis When Starting With a Good Solution , 2020, PPSN.

[16]  Benjamin Doerr,et al.  Runtime Analysis of a Heavy-Tailed (1+(λ, λ)) Genetic Algorithm on Jump Functions , 2020, PPSN.

[17]  Benjamin Doerr,et al.  The (1 + (λ,λ)) GA is even faster on multimodal problems , 2020, GECCO.

[18]  Benjamin Doerr,et al.  Fixed-Target Runtime Analysis , 2020, Algorithmica.

[19]  Benjamin Doerr.  Does Comma Selection Help to Cope with Local Optima? , 2020, GECCO.

[20]  Benjamin Doerr,et al.  Multiplicative Up-Drift , 2019, Algorithmica.

[21]  Per Kristian Lehre,et al.  Parallel Black-Box Complexity With Tail Bounds , 2019, IEEE Transactions on Evolutionary Computation.

[22]  Benjamin Doerr,et al.  Significance-Based Estimation-of-Distribution Algorithms , 2018, IEEE Transactions on Evolutionary Computation.

[23]  Benjamin Doerr,et al.  Runtime Analysis for Self-adaptive Mutation Rates , 2018, Algorithmica.

[24]  Pietro Simone Oliveto,et al.  When Hypermutations and Ageing Enable Artificial Immune Systems to Outperform Evolutionary Algorithms , 2018, Theor. Comput. Sci..

[25]  Benjamin Doerr,et al.  Probabilistic Tools for the Analysis of Randomized Optimization Heuristics , 2018, Theory of Evolutionary Computation.

[26]  Carola Doerr,et al.  Complexity Theory for Discrete Black-Box Optimization Heuristics , 2018, Theory of Evolutionary Computation.

[27]  Thomas Bäck,et al.  Theory of Evolutionary Computation: Recent Developments in Discrete Optimization , 2020, Theory of Evolutionary Computation.

[28]  Benjamin Doerr,et al.  A tight runtime analysis for the (1 + (λ, λ)) GA on leadingones , 2019, FOGA '19.

[29]  Per Kristian Lehre,et al.  On the limitations of the univariate marginal distribution algorithm to deception and where bivariate EDAs might help , 2019, FOGA '19.

[30]  Andrei Lissovoi,et al.  On the Time Complexity of Algorithm Selection Hyper-Heuristics for Multimodal Optimisation , 2019, AAAI.

[31]  Benjamin Doerr,et al.  Analyzing randomized search heuristics via stochastic domination , 2019, Theor. Comput. Sci..

[32]  Benjamin Doerr,et al.  A tight runtime analysis for the cGA on jump functions: EDAs can cross fitness valleys at no extra cost , 2019, GECCO.

[33]  Kurt Mehlhorn,et al.  The Query Complexity of a Permutation-Based Variant of Mastermind , 2019, Discret. Appl. Math..

[34]  Benjamin Doerr,et al.  The ($$1+\lambda $$1+λ) Evolutionary Algorithm with Self-Adjusting Mutation Rate , 2018, Algorithmica.

[35]  C. Witt,et al.  On the Choice of the Update Strength in Estimation-of-Distribution Algorithms and Ant Colony Optimization , 2018, Algorithmica.

[36]  P. Lehre,et al.  Level-Based Analysis of the Univariate Marginal Distribution Algorithm , 2018, Algorithmica.

[37]  Benjamin Doerr,et al.  A tight runtime analysis for the (μ + λ) EA , 2018, GECCO.

[38]  Andrew M. Sutton,et al.  On the runtime dynamics of the compact genetic algorithm on jump functions , 2018, GECCO.

[39]  Per Kristian Lehre,et al.  University of Birmingham Level-based analysis of the population-based incremental learning algorithm , 2018 .

[40]  Per Kristian Lehre,et al.  Escaping Local Optima Using Crossover With Emergent Diversity , 2018, IEEE Transactions on Evolutionary Computation.

[41]  Angelika Steger,et al.  Drift Analysis and Evolutionary Algorithms Revisited , 2016, Combinatorics, Probability and Computing.

[42]  Dirk Sudholt,et al.  How to Escape Local Optima in Black Box Optimisation: When Non-elitism Outperforms Elitism , 2017, Algorithmica.

[43]  Maxim Buzdalov,et al.  Evaluation of heavy-tailed mutation operator on maximum flow test generation problem , 2017, GECCO.

[44]  Dirk Sudholt,et al.  Principled Design and Runtime Analysis of Abstract Convex Evolutionary Search , 2017, Evolutionary Computation.

[45]  Benjamin Doerr,et al.  Runtime analysis of the (1 + (λ, λ)) genetic algorithm on random satisfiable 3-CNF formulas , 2017, GECCO.

[46]  Benjamin Doerr,et al.  The (1+λ) evolutionary algorithm with self-adjusting mutation rate , 2017, GECCO.

[47]  Benjamin Doerr,et al.  Fast genetic algorithms , 2017, GECCO.

[48]  Carola Doerr,et al.  Introducing Elitist Black-Box Models: When Does Elitist Behavior Weaken the Performance of Evolutionary Algorithms? , 2015, Evolutionary Computation.

[49]  ON TWO PROBLEMS OF INFORMATION THEORY , 2017 .

[50]  Frank Neumann,et al.  Time Complexity Analysis of Evolutionary Algorithms on Random Satisfiable k-CNF Formulas , 2016, Algorithmica.

[51]  Tobias Friedrich,et al.  EDAs cannot be Balanced and Stable , 2016, GECCO.

[52]  Benjamin Doerr,et al.  Optimal Parameter Choices via Precise Black-Box Analysis , 2016, GECCO.

[53]  Duc-Cuong Dang,et al.  Escaping Local Optima with Diversity Mechanisms and Crossover , 2016, GECCO.

[54]  Carola Doerr,et al.  The (1+1) Elitist Black-Box Complexity of LeadingOnes , 2016, GECCO.

[55]  Dirk Sudholt,et al.  Towards a Runtime Comparison of Natural and Artificial Evolution , 2015, Algorithmica.

[56]  Duc-Cuong Dang,et al.  Level-Based Analysis of Genetic Algorithms and Other Search Processes , 2014, bioRxiv.

[57]  Duc-Cuong Dang,et al.  Runtime Analysis of Non-elitist Populations: From Classical Optimisation to Partial Information , 2016, Algorithmica.

[58]  Benjamin Doerr,et al.  From black-box complexity to designing new genetic algorithms , 2015, Theor. Comput. Sci..

[59]  Kenneth Sörensen,et al.  Metaheuristics - the metaphor exposed , 2015, Int. Trans. Oper. Res..

[60]  Per Kristian Lehre,et al.  Unbiased Black-Box Complexity of Parallel Search , 2014, PPSN.

[61]  Dirk Sudholt,et al.  The choice of the offspring population size in the (1, λ) evolutionary algorithm , 2014, Theor. Comput. Sci..

[62]  William F. Punch,et al.  Parameter-less population pyramid , 2014, GECCO.

[63]  Pietro Simone Oliveto,et al.  On the runtime analysis of stochastic ageing mechanisms , 2014, GECCO.

[64]  Dirk Sudholt,et al.  A New Method for Lower Bounds on the Running Time of Evolutionary Algorithms , 2011, IEEE Transactions on Evolutionary Computation.

[65]  Frank Neumann,et al.  Bioinspired computation in combinatorial optimization: algorithms and their computational complexity , 2010, GECCO '12.

[66]  Thomas Jansen,et al.  Analyzing Evolutionary Algorithms: The Computer Science Perspective , 2012 .

[67]  Per Kristian Lehre,et al.  Drift analysis , 2012, GECCO '12.

[68]  Stephen R. Marsland,et al.  Convergence Properties of (μ + λ) Evolutionary Algorithms , 2011, AAAI.

[69]  Per Kristian Lehre,et al.  Fitness-levels for non-elitist populations , 2011, GECCO '11.

[70]  Benjamin Doerr,et al.  Black-box complexities of combinatorial problems , 2011, GECCO '11.

[71]  Dirk Sudholt,et al.  Runtime analysis of the 1-ANT ant colony optimizer , 2011, Theor. Comput. Sci..

[72]  Per Kristian Lehre,et al.  Faster black-box algorithms through higher arity operators , 2010, FOGA '11.

[73]  Benjamin Doerr,et al.  Optimal Fixed and Adaptive Mutation Rates for the LeadingOnes Problem , 2010, PPSN.

[74]  Per Kristian Lehre,et al.  Negative Drift in Populations , 2010, PPSN.

[75]  Per Kristian Lehre,et al.  Black-Box Search by Unbiased Variation , 2010, GECCO '10.

[76]  Peter Sanders.  Algorithm Engineering , 2010, Informatik-Spektrum.

[77]  Pietro Simone Oliveto,et al.  Theoretical analysis of fitness-proportional selection: landscapes and efficiency , 2009, GECCO.

[78]  Karsten Weicker,et al.  Metropolis and Symmetric Functions: A Swan Song , 2009, EvoCOP.

[79]  R. Paul Wiegand,et al.  Black-box search by elimination of fitness functions , 2009, FOGA '09.

[80]  Dirk Sudholt,et al.  Analysis of different MMAS ACO algorithms on unimodal functions and plateaus , 2009, Swarm Intelligence.

[81]  Frank Neumann,et al.  Rigorous analyses of fitness-proportional selection for optimizing linear functions , 2008, GECCO '08.

[82]  Peter Rossmanith,et al.  Local search algorithms , 2008 .

[83]  Howard Straubing,et al.  Theory of Computing Systems , 2008 .

[84]  Klaus Meer,et al.  Simulated Annealing versus Metropolis for a TSP instance , 2007, Inf. Process. Lett..

[85]  Thomas Jansen,et al.  A comparison of simulated annealing with a simple evolutionary algorithm on pseudo-boolean functions of unitation , 2007, Theor. Comput. Sci..

[86]  Jens Jägersküpper,et al.  When the Plus Strategy Outperforms the Comma Strategyand When Not , 2007, 2007 IEEE Symposium on Foundations of Computational Intelligence.

[87]  Olivier Teytaud,et al.  On the Ultimate Convergence Rates for Isotropic Algorithms and the Best Choices Among Various Forms of Isotropy , 2006, PPSN.

[88]  Dan Suciu,et al.  Journal of the ACM , 2006 .

[89]  Carsten Witt,et al.  Runtime Analysis of the ( + 1) EA on Simple Pseudo-Boolean Functions , 2006, Evolutionary Computation.

[90]  Kenneth A. De Jong,et al.  Design and Management of Complex Technical Processes and Systems by Means of Computational Intelligence Methods on the Choice of the Offspring Population Size in Evolutionary Algorithms on the Choice of the Offspring Population Size in Evolutionary Algorithms , 2004 .

[91]  Ingo Wegener,et al.  Simulated Annealing Beats Metropolis in Combinatorial Optimization , 2005, ICALP.

[92]  Thomas Jansen,et al.  UNIVERSITY OF DORTMUND REIHE COMPUTATIONAL INTELLIGENCE COLLABORATIVE RESEARCH CENTER 531 Design and Management of Complex Technical Processes and Systems by means of Computational Intelligence Methods Upper and Lower Bounds for Randomized Search Heuristics in Black-Box Optimization , 2004 .

[93]  Xin Yao,et al.  A study of drift analysis for estimating computation time of evolutionary algorithms , 2004, Natural Computing.

[94]  Ingo Wegener,et al.  Evolutionary Algorithms and the Maximum Matching Problem , 2003, STACS.

[95]  J. van Leeuwen,et al.  Theoretical Computer Science , 2003, Lecture Notes in Computer Science.

[96]  Thomas Jansen,et al.  A New Framework for the Valuation of Algorithms for Black-Box Optimization , 2002, FOGA.

[97]  Thomas Jansen,et al.  On the analysis of the (1+1) evolutionary algorithm , 2002, Theor. Comput. Sci..

[98]  Xin Yao,et al.  Drift analysis and average time complexity of evolutionary algorithms , 2001, Artif. Intell..

[99]  Mark Jerrum,et al.  The Metropolis Algorithm for Graph Bisection , 1998, Discret. Appl. Math..

[100]  N. Fisher,et al.  Probability Inequalities for Sums of Bounded Random Variables , 1994 .

[101]  Peter L. Hammer,et al.  Discrete Applied Mathematics , 1993 .

[102]  Heinz Mühlenbein,et al.  How Genetic Algorithms Really Work: Mutation and Hillclimbing , 1992, PPSN.

[103]  Bruce E. Hajek,et al.  The time complexity of maximum matching by simulated annealing , 1988, JACM.

[104]  N. Metropolis,et al.  Equation of State Calculations by Fast Computing Machines , 1953, Resonance.

[105]  H. Chernoff.  A Measure of Asymptotic Efficiency for Tests of a Hypothesis Based on the sum of Observations , 1952 .

[106]  Journal of Chemical Physics , 1932, Nature.

[107]  Schloss Birlinghoven,et al.  How Genetic Algorithms Really Work I.mutation and Hillclimbing , 2022 .