Compiling a benchmarking test-suite for combinatorial black-box optimization: a position paper

This contribution focuses on the challenge of formulating a set of benchmark problems and/or a test-suite for Combinatorial Optimization problems when treated as black-box global optimization problems. We discuss the involved dilemmas and possible obstacles of such a compilation. To this end, we formulate a list of design questions that need to be answered as a first step in this compilation process. We articulate our perspective on these questions by proposing a rough classification of relevant problem classes, answering the posed questions, and suggesting a preliminary set of problems. While this position paper addresses the Evolutionary Computation community, it intends to offer an open-minded Computer Science perspective - by considering the broad definition of Combinatorial Optimization and by accounting for equivalent threads within Operations Research and Mathematical Programming communities. At the same time, this work capitalizes on prior art in algorithms' benchmarking, including the authors' own experience with the continuous BBOB benchmark problem set, as well as a number of discrete black-box optimization challenges frequently encountered in practice.

[1]  Benjamin Doerr,et al.  Playing Mastermind With Many Colors , 2012, SODA.

[2]  Anne Auger,et al.  Comparing results of 31 algorithms from the black-box optimization benchmarking BBOB-2009 , 2010, GECCO '10.

[3]  Robert E. Bixby,et al.  MIP: Theory and Practice - Closing the Gap , 1999, System Modelling and Optimization.

[4]  Anne Auger,et al.  Real-Parameter Black-Box Optimization Benchmarking 2009: Noiseless Functions Definitions , 2009 .

[5]  Khaled Ghédira,et al.  Constraint Satisfaction Problems: Ghédira/Constraint Satisfaction Problems , 2013 .

[6]  Kenneth Steiglitz,et al.  Combinatorial Optimization: Algorithms and Complexity , 1981 .

[7]  R. Burkard.  Quadratic Assignment Problems , 1984 .

[8]  Anne Auger,et al.  COCO: a platform for comparing continuous optimizers in a black-box setting , 2016, Optim. Methods Softw..

[9]  Thomas Bäck,et al.  Evolutionary algorithms in theory and practice - evolution strategies, evolutionary programming, genetic algorithms , 1996 .

[10]  Hsien-Kuei Hwang,et al.  Probabilistic Analysis of the (1+1)-Evolutionary Algorithm , 2014, Evolutionary Computation.

[11]  Dogan Corus,et al.  Standard Steady State Genetic Algorithms Can Hillclimb Faster Than Mutation-Only Evolutionary Algorithms , 2017, IEEE Transactions on Evolutionary Computation.

[12]  Pietro Simone Oliveto,et al.  Standard steady state genetic algorithms can hillclimb faster than evolutionary algorithms using standard bit mutation , 2018, GECCO.

[13]  Benjamin Doerr,et al.  Static and Self-Adjusting Mutation Strengths for Multi-valued Decision Variables , 2018, Algorithmica.

[14]  Edward Rothberg,et al.  An Evolutionary Algorithm for Polishing Mixed Integer Programming Solutions , 2007, INFORMS J. Comput..

[15]  Cristopher Moore,et al.  The Nature of Computation , 2011 .

[16]  Benjamin Doerr,et al.  Optimal Static and Self-Adjusting Parameter Choices for the (1+(λ,λ))\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$( , 2017, Algorithmica.

[17]  L. Darrell Whitley,et al.  Evaluating Evolutionary Algorithms , 1996, Artif. Intell..

[18]  L. Darrell Whitley,et al.  Next generation genetic algorithms , 2017, GECCO.

[19]  Benjamin Doerr,et al.  Edge-based representation beats vertex-based representation in shortest path problems , 2010, GECCO '10.

[20]  Devika Subramanian,et al.  Random 3-SAT: The Plot Thickens , 2000, CP.

[21]  Carola Doerr,et al.  Complexity Theory for Discrete Black-Box Optimization Heuristics , 2018, Theory of Evolutionary Computation.

[22]  Anne Auger,et al.  Theory of Randomized Search Heuristics: Foundations and Recent Developments , 2011, Theory of Randomized Search Heuristics.

[23]  M. R. Rao,et al.  Combinatorial Optimization , 1992, NATO ASI Series.

[24]  Benjamin Doerr,et al.  k-Bit Mutation with Self-Adjusting k Outperforms Standard Bit Mutation , 2016, PPSN.

[25]  Carsten Witt,et al.  The Interplay of Population Size and Mutation Probability in the (1+λ) EA on OneMax , 2017, Algorithmica.

[26]  Christian Blum,et al.  Hybrid Metaheuristics: Powerful Tools for Optimization , 2016 .

[27]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[28]  Stephan Mertens,et al.  Low autocorrelation binary sequences , 2015, 1512.02475.

[29]  Kiyoshi Tanaka,et al.  Performance and Scalability of Genetic Algorithms on NK-Landscapes , 2008, Recent Advances in Evolutionary Computation for Combinatorial Optimization.

[30]  Witold Pedrycz,et al.  Springer Handbook of Computational Intelligence , 2015, Springer Handbook of Computational Intelligence.

[31]  D. Ackley.  A connectionist machine for genetic hillclimbing , 1987 .

[32]  Anne Auger,et al.  COCO: Performance Assessment , 2016, ArXiv.

[33]  Carsten Witt,et al.  The Interplay of Population Size and Mutation Probability in the ($$1+\lambda $$1+λ) EA on OneMax , 2016, Algorithmica.

[34]  David B. Fogel,et al.  Evolutionary algorithms in theory and practice , 1997, Complex.

[35]  Benjamin Doerr,et al.  Unbiased Black-Box Complexities of Jump Functions , 2014, Evolutionary Computation.

[36]  Kurt Mehlhorn,et al.  The Query Complexity of Finding a Hidden Permutation , 2013, Space-Efficient Data Structures, Streams, and Algorithms.

[37]  E. D. Weinberger,et al.  The NK model of rugged fitness landscapes and its application to maturation of the immune response. , 1989, Journal of theoretical biology.

[38]  Benjamin Doerr,et al.  Optimal Parameter Choices via Precise Black-Box Analysis , 2016, GECCO.

[39]  Paul Erdgs,et al.  ON TWO PROBLEMS OF INFORMATION THEORY bY PAUL ERDGS and ALFRJ~D RgNYI , 2001 .

[40]  I. A. Pasha,et al.  Bi-alphabetic pulse compression radar signal design , 2000 .

[41]  Marvin Künnemann,et al.  Optimizing linear functions with the (1+λ) evolutionary algorithm - Different asymptotic runtimes for different instances , 2015, Theor. Comput. Sci..

[42]  Dieter Beule,et al.  Evolutionary search for low autocorrelated binary sequences , 1998, IEEE Trans. Evol. Comput..

[43]  R. P. Ingalls,et al.  FOURTH TEST OF GENERAL RELATIVITY: PRELIMINARY RESULTS. , 1968 .

[44]  Per Kristian Lehre,et al.  Black-Box Search by Unbiased Variation , 2010, GECCO '10.