Analyzing the BBOB Results by Means of Benchmarking Concepts

We present methods to answer two basic questions that arise when benchmarking optimization algorithms. The first one is: which algorithm is the “best” one? and the second one is: which algorithm should I use for my real-world problem? Both are connected and neither is easy to answer. We present a theoretical framework for designing and analyzing the raw data of such benchmark experiments. This represents a first step in answering the aforementioned questions. The 2009 and 2010 BBOB benchmark results are analyzed by means of this framework and we derive insight regarding the answers to the two questions. Furthermore, we discuss how to properly aggregate rankings from algorithm evaluations on individual problems into a consensus, its theoretical background and which common pitfalls should be avoided. Finally, we address the grouping of test problems into sets with similar optimizer rankings and investigate whether these are reflected by already proposed test problem characteristics, finding that this is not always the case.

[1]  Wade D. Cook,et al.  Ordinal Information and Preference Structures: Decision Models and Applications , 1992 .

[2]  Ali S. Hadi,et al.  Finding Groups in Data: An Introduction to Chster Analysis , 1991 .

[3]  Donald G. Saari,et al.  A geometric examination of Kemeny's rule , 2000, Soc. Choice Welf..

[4]  Heike Trautmann,et al.  Benchmarking evolutionary multiobjective optimization algorithms , 2010, IEEE Congress on Evolutionary Computation.

[5]  Anne Auger,et al.  Performance evaluation of an advanced local search evolutionary algorithm , 2005, 2005 IEEE Congress on Evolutionary Computation.

[6]  Anne Auger,et al.  BBOB 2009: Comparison Tables of All Algorithms on All Noiseless Functions , 2010 .

[7]  K. Arrow.  A Difficulty in the Concept of Social Welfare , 1950, Journal of Political Economy.

[8]  Heike Trautmann,et al.  On the Distribution of EMOA Hypervolumes , 2010, LION.

[9]  Anne Auger,et al.  Real-Parameter Black-Box Optimization Benchmarking 2009: Noiseless Functions Definitions , 2009 .

[10]  Kurt Hornik,et al.  Deriving Consensus Rankings from Benchmarking Experiments , 2006, GfKl.

[11]  Franklin A. Graybill,et al.  Introduction to the Theory of Statistics, 3rd ed. , 1974 .

[12]  João Gama,et al.  Characterizing the Applicability of Classification Algorithms Using Meta-Level Learning , 1994, ECML.

[13]  John G. Kemeny,et al.  Mathematical models in the social sciences , 1964 .

[14]  Bernd Bischl,et al.  Exploratory landscape analysis , 2011, GECCO '11.

[15]  David J. Hunter.  Essentials of Discrete Mathematics , 2008 .

[16]  Heike Trautmann,et al.  Benchmarking Evolutionary Algorithms: Towards Exploratory Landscape Analysis , 2010, PPSN.

[17]  Raymond Ros,et al.  Real-Parameter Black-Box Optimization Benchmarking 2009: Experimental Setup , 2009 .

[18]  Eric R. Ziegel,et al.  The Elements of Statistical Learning , 2003, Technometrics.

[19]  Olivier Teytaud,et al.  Continuous lunches are free! , 2007, GECCO '07.

[20]  J. Wolfowitz,et al.  An Introduction to the Theory of Statistics , 1951, Nature.

[21]  Wei-Yin Loh,et al.  Classification and regression trees , 2011, WIREs Data Mining Knowl. Discov..

[22]  Lothar Thiele,et al.  Multiobjective Optimization Using Evolutionary Algorithms - A Comparative Case Study , 1998, PPSN.

[23]  Anne Auger,et al.  Comparing results of 31 algorithms from the black-box optimization benchmarking BBOB-2009 , 2010, GECCO '10.

[24]  John G. Kemeny,et al.  Mathematical models in the social sciences , 1964 .