Benchmarking large-scale continuous optimizers: The bbob-largescale testbed, a COCO software guide and beyond

Abstract Benchmarking of optimization solvers is an important and compulsory task for performance assessment that in turn can help in improving the design of algorithms. It is a repetitive and tedious task. Yet, this task has been greatly automatized in the past ten years with the development of the Comparing Continuous Optimizers platform ( COCO ). In this context, this paper presents a new testbed, called bbob-largescale , that contains functions ranging from dimension 20 to 640, compatible with and extending the well-known single-objective noiseless bbob test suite to larger dimensions. The test suite contains 24 single-objective functions in continuous domain, built to model well-known difficulties in continuous optimization and to test the scaling behavior of algorithms. To reduce the computational demand of the orthogonal search space transformations that appear in the bbob test suite, while retaining some desired properties, we use permuted block diagonal orthogonal matrices. The paper discusses implementation technicalities and presents a guide for using the test suite within the COCO  platform and for interpreting the postprocessed output. The source code of the new test suite is available on GitHub as part of the open source COCO  benchmarking platform.

[1]  Nicholas I. M. Gould,et al.  CUTE: constrained and unconstrained testing environment , 1995, TOMS.

[2]  Ilya Loshchilov,et al.  LM-CMA: An Alternative to L-BFGS for Large-Scale Black Box Optimization , 2015, Evolutionary Computation.

[3]  Francisco Herrera,et al.  Editorial scalability of evolutionary algorithms and other metaheuristics for large-scale continuous optimization problems , 2011, Soft Comput..

[4]  Xiaodong Li,et al.  Benchmark Functions for the CEC'2010 Special Session and Competition on Large-Scale , 2009 .

[5]  Anne Auger,et al.  Comparing results of 31 algorithms from the black-box optimization benchmarking BBOB-2009 , 2010, GECCO '10.

[6]  Konstantinos Varelas,et al.  Benchmarking multivariate solvers of SciPy on the noiseless testbed , 2019, GECCO.

[7]  Anne Auger,et al.  COCO: Performance Assessment , 2016, ArXiv.

[8]  Anne Auger,et al.  Permuted Orthogonal Block-Diagonal Transformation Matrices for Large Scale Optimization Benchmarking , 2016, GECCO.

[9]  Nikolaus Hansen,et al.  Completely Derandomized Self-Adaptation in Evolution Strategies , 2001, Evolutionary Computation.

[10]  Anne Auger,et al.  A Comparative Study of Large-Scale Variants of CMA-ES , 2018, PPSN.

[11]  Qingfu Zhang,et al.  A Simple Yet Efficient Evolution Strategy for Large-Scale Black-Box Optimization , 2018, IEEE Transactions on Evolutionary Computation.

[12]  Petr Posík,et al.  JADE, an adaptive differential evolution algorithm, benchmarked on the BBOB noiseless testbed , 2012, GECCO '12.

[13]  Alex S. Fukunaga,et al.  Tuning differential evolution for cheap, medium, and expensive computational budgets , 2015, 2015 IEEE Congress on Evolutionary Computation (CEC).

[14]  Anne Auger,et al.  COCO: a platform for comparing continuous optimizers in a black-box setting , 2016, Optim. Methods Softw..

[15]  Jorge Nocedal,et al.  On the limited memory BFGS method for large scale optimization , 1989, Math. Program..

[16]  Anne Auger,et al.  COCO: The Experimental Procedure , 2016, ArXiv.

[17]  Nikolaus Hansen,et al.  Benchmarking of Continuous Black Box Optimization Algorithms , 2012, Evolutionary Computation.

[18]  F. Mezzadri.  How to generate random matrices from the classical compact groups , 2006, math-ph/0609050.

[19]  Stefan M. Wild,et al.  Benchmarking Derivative-Free Optimization Algorithms , 2009, SIAM J. Optim..

[20]  R. Salomon.  Re-evaluating genetic algorithm performance under coordinate rotation of benchmark functions. A survey of some theoretical and practical aspects of genetic algorithms. , 1996, Bio Systems.

[21]  Ata Kabán,et al.  Toward Large-Scale Continuous EDA: A Random Matrix Theory Perspective , 2013, Evolutionary Computation.

[22]  Nicholas I. M. Gould,et al.  CUTEst: a Constrained and Unconstrained Testing Environment with safe threads for mathematical optimization , 2013, Computational Optimization and Applications.

[23]  Youhei Akimoto,et al.  Online Model Selection for Restricted Covariance Matrix Adaptation , 2016, PPSN.