Investigating benchmark correlations when comparing algorithms with parameter tuning

Benchmarks are important for comparing performance of optimisation algorithms, but we can select instances that present our algorithm favourably, and dismiss those on which our algorithm under-performs. Also related are automated design of algorithms, which use problem instances (benchmarks) to train an algorithm: careful choice of instances is needed for the algorithm to generalise. We sweep parameter settings of differential evolution to applied to the BBOB benchmarks. Several benchmark functions are highly correlated. This may lead to the false conclusion that an algorithm performs well in general, when it performs poorly on a few key instances. These correlations vary with the number of evaluations.

[1]  Alex S. Fukunaga,et al.  Tuning differential evolution for cheap, medium, and expensive computational budgets , 2015, 2015 IEEE Congress on Evolutionary Computation (CEC).

[2]  Anne Auger,et al.  Comparing results of 31 algorithms from the black-box optimization benchmarking BBOB-2009 , 2010, GECCO '10.

[3]  Yuri Malitsky,et al.  Features for Exploiting Black-Box Optimization Problem Structure , 2013, LION.

[4]  Rainer Storn,et al.  Differential Evolution – A Simple and Efficient Heuristic for global Optimization over Continuous Spaces , 1997, J. Glob. Optim..

[5]  David S. Johnson,et al.  A theoretician's guide to the experimental analysis of algorithms , 1999, Data Structures, Near Neighbor Searches, and Methodology.

[6]  Alexander Mendiburu,et al.  Are we generating instances uniformly at random? , 2017, 2017 IEEE Congress on Evolutionary Computation (CEC).

[7]  Bernd Bischl,et al.  Exploratory landscape analysis , 2011, GECCO '11.

[8]  David H. Wolpert,et al.  No free lunch theorems for optimization , 1997, IEEE Trans. Evol. Comput..

[9]  Anne Auger,et al.  Real-Parameter Black-Box Optimization Benchmarking 2009: Noiseless Functions Definitions , 2009 .

[10]  Roberto Santana,et al.  Multi-objective NM-Landscapes , 2015, GECCO.

[11]  Kate Smith-Miles,et al.  Performance Analysis of Continuous Black-Box Optimization Algorithms via Footprints in Instance Space , 2016, Evolutionary Computation.

[12]  Mario A. Muñoz,et al.  A Meta-learning Prediction Model of Algorithm Performance for Continuous Optimization Problems , 2012, PPSN.

[13]  Marc Schoenauer,et al.  Feature Based Algorithm Configuration: A Case Study with Differential Evolution , 2016, PPSN.

[14]  Wojciech Jaskowski,et al.  Better GP benchmarks: community survey results and proposals , 2012, Genetic Programming and Evolvable Machines.

[15]  Saman K. Halgamuge,et al.  Exploratory Landscape Analysis of Continuous Space Optimization Problems Using Information Content , 2015, IEEE Transactions on Evolutionary Computation.

[16]  Leslie Pérez Cáceres,et al.  The irace package: Iterated racing for automatic algorithm configuration , 2016 .

[17]  Nelishia Pillay,et al.  Evolving hyper-heuristics for the uncapacitated examination timetabling problem , 2012, J. Oper. Res. Soc..

[18]  Jerry Swan,et al.  Template method hyper-heuristics , 2014, GECCO.

[19]  Riccardo Poli,et al.  Evolving Problems to Learn About Particle Swarm Optimizers and Other Search Algorithms , 2006, IEEE Transactions on Evolutionary Computation.

[20]  Kate Smith-Miles,et al.  Generating new test instances by evolving in instance space , 2015, Comput. Oper. Res..

[21]  Mengjie Zhang,et al.  Automated Design of Production Scheduling Heuristics: A Review , 2016, IEEE Transactions on Evolutionary Computation.

[22]  John A. W. McCall,et al.  Generating Easy and Hard Problems using the Proximate Optimality Principle , 2015, GECCO.

[23]  Francisco J. Rodríguez,et al.  Arbitrary function optimisation with metaheuristics , 2012, Soft Comput..

[24]  Bernd Bischl,et al.  Algorithm selection based on exploratory landscape analysis and cost-sensitive learning , 2012, GECCO '12.

[25]  Marc Schoenauer,et al.  Per instance algorithm configuration of CMA-ES with limited budget , 2017, GECCO.

[26]  John A. W. McCall,et al.  Interpolated continuous optimisation problems with tunable landscape features , 2017, GECCO.

[27]  Bernd Bischl,et al.  Analyzing the BBOB Results by Means of Benchmarking Concepts , 2015, Evolutionary Computation.