On constructing alternative benchmark suite for evolutionary algorithms

Abstract Benchmark testing offers performance measurement for an evolutionary algorithm before it is put into applications. In this paper, a systematic method to construct a benchmark test suite is proposed. A set of established algorithms are employed. For each algorithm, a uniquely easy problem instance is generated by evolution. The resulting instances consist of a novel benchmark test suite. Each problem instance is favorable (uniquely easy) to one algorithm only. A hierarchical fitness assignment method, which is based on statistical test results, is designed to generate uniquely easy (or hard) problem instances for an algorithm. Experimental results show that each algorithm performs the best robustly on its uniquely favorable problem. The testing results are repeatable. The distribution of algorithm performance in the suite is unbiased (or uniform), which mimics any subset of real-world problems that is uniformly distributed. The resulting suite offers 1) an alternative benchmark suite to evolutionary algorithms; 2) a novel method of assessing novel algorithms; and 3) meaningful training and testing problems for evolutionary algorithm selectors and portfolios.

[1]  Jano I. van Hemert,et al.  Evolving Combinatorial Problem Instances That Are Difficult to Solve , 2006, Evolutionary Computation.

[2]  Francisco Herrera,et al.  Since CEC 2005 competition on real-parameter optimisation: a decade of research, progress and comparative analysis’s weakness , 2017, Soft Comput..

[3]  Mark Hoogendoorn,et al.  Parameter Control in Evolutionary Algorithms: Trends and Challenges , 2015, IEEE Transactions on Evolutionary Computation.

[4]  Jing J. Liang,et al.  Problem Definitions and Evaluation Criteria for the CEC 2005 Special Session on Real-Parameter Optimization , 2005 .

[5]  H. B. Mann,et al.  On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other , 1947 .

[6]  Alexander Mendiburu,et al.  A Tunable Generator of Instances of Permutation-Based Combinatorial Optimization Problems , 2016, IEEE Transactions on Evolutionary Computation.

[7]  Xin Yao,et al.  Population-based Algorithm Portfolios with automated constituent algorithms selection , 2014, Inf. Sci..

[8]  Alex S. Fukunaga,et al.  Improving the search performance of SHADE using linear population size reduction , 2014, 2014 IEEE Congress on Evolutionary Computation (CEC).

[9]  Xiaodong Li,et al.  A framework for generating tunable test functions for multimodal optimization , 2011, Soft Comput..

[10]  Jason Sheng-Hong Tsai,et al.  A self-optimization approach for L-SHADE incorporated with eigenvector-based crossover and successful-parent-selecting framework on CEC 2015 benchmark set , 2015, 2015 IEEE Congress on Evolutionary Computation (CEC).

[11]  Margaret J. Eppstein,et al.  Tunably Rugged Landscapes With Known Maximum and Minimum , 2014, IEEE Transactions on Evolutionary Computation.

[12]  Rainer Storn,et al.  Differential Evolution – A Simple and Efficient Heuristic for global Optimization over Continuous Spaces , 1997, J. Glob. Optim..

[13]  Xin Yao,et al.  Evolutionary programming made faster , 1999, IEEE Trans. Evol. Comput..

[14]  Yang Lou,et al.  A Sequential Learnable Evolutionary Algorithm with a Novel Knowledge Base Generation Method , 2017, SEAL.

[15]  David H. Wolpert,et al.  No free lunch theorems for optimization , 1997, IEEE Trans. Evol. Comput..

[16]  John N. Hooker,et al.  Testing heuristics: We have it all wrong , 1995, J. Heuristics.

[17]  Jano I. van Hemert,et al.  Discovering the suitability of optimisation algorithms by learning from evolved instances , 2011, Annals of Mathematics and Artificial Intelligence.

[18]  Ruhul A. Sarker,et al.  Testing united multi-operator evolutionary algorithms-II on single objective optimization problems , 2016, 2016 IEEE Congress on Evolutionary Computation (CEC).

[19]  Kate Smith-Miles,et al.  Towards objective measures of algorithm performance across instance space , 2014, Comput. Oper. Res..

[20]  W. Kruskal,et al.  Use of Ranks in One-Criterion Variance Analysis , 1952 .

[21]  Mauricio Zambrano-Bigiarini,et al.  Standard Particle Swarm Optimisation 2011 at CEC-2013: A baseline for future PSO improvements , 2013, 2013 IEEE Congress on Evolutionary Computation.

[22]  Jürgen Branke,et al.  Evolutionary optimization in uncertain environments-a survey , 2005, IEEE Transactions on Evolutionary Computation.

[23]  Qingfu Zhang,et al.  Differential Evolution With Composite Trial Vector Generation Strategies and Control Parameters , 2011, IEEE Transactions on Evolutionary Computation.

[24]  Robert G. Reynolds,et al.  An ensemble sinusoidal parameter adaptation incorporated with L-SHADE for solving CEC2014 benchmark problems , 2016, 2016 IEEE Congress on Evolutionary Computation (CEC).

[25]  Riccardo Poli,et al.  Evolving Problems to Learn About Particle Swarm Optimizers and Other Search Algorithms , 2006, IEEE Transactions on Evolutionary Computation.

[26]  Ilya Loshchilov,et al.  CMA-ES with restarts for solving CEC 2013 benchmark problems , 2013, 2013 IEEE Congress on Evolutionary Computation.

[27]  Jing J. Liang,et al.  Novel benchmark functions for continuous multimodal optimization with comparative results , 2016, Swarm Evol. Comput..

[28]  Xin Yao,et al.  On the Easiest and Hardest Fitness Functions , 2012, IEEE Transactions on Evolutionary Computation.

[29]  Yang Lou,et al.  Sequential Learnable Evolutionary Algorithm: A Research Program , 2015, 2015 IEEE International Conference on Systems, Man, and Cybernetics.

[30]  Yang Lou,et al.  Which algorithm should I choose: An evolutionary algorithm portfolio approach , 2016, Appl. Soft Comput..

[31]  Jeffrey S. Buzas,et al.  An Analysis of $N\!K$ Landscapes: Interaction Structure, Statistical Properties, and Expected Number of Local Optima , 2014, IEEE Transactions on Evolutionary Computation.

[32]  Mario A. Muñoz,et al.  ICARUS: Identification of complementary algorithms by uncovered sets , 2016, 2016 IEEE Congress on Evolutionary Computation (CEC).

[33]  Ruhul A. Sarker,et al.  GA with a new multi-parent crossover for constrained optimization , 2011, 2011 IEEE Congress of Evolutionary Computation (CEC).

[34]  M. Fay,et al.  Wilcoxon-Mann-Whitney or t-test? On assumptions for hypothesis tests and multiple interpretations of decision rules. , 2010, Statistics surveys.

[35]  Shiu Yin Yuen,et al.  On composing an algorithm portfolio , 2015, Memetic Computing.

[36]  Kate Smith-Miles,et al.  Measuring instance difficulty for combinatorial optimization problems , 2012, Comput. Oper. Res..

[37]  Jürgen Branke,et al.  Evolutionary search for difficult problem instances to support the design of job shop dispatching rules , 2011, Eur. J. Oper. Res..

[38]  Jun Zhang,et al.  Benchmarking Stochastic Algorithms for Global Optimization Problems by Visualizing Confidence Intervals , 2017, IEEE Transactions on Cybernetics.

[39]  Jason H. Moore,et al.  Erratum to: Evolving hard problems: generating human genetics datasets with a complex etiology , 2016, BioData Mining.

[40]  Marcus Gallagher,et al.  A general-purpose tunable landscape generator , 2006, IEEE Transactions on Evolutionary Computation.

[41]  Tomoharu Nagao,et al.  Evolving search spaces to emphasize the performance difference of real-coded crossovers using genetic programming , 2010, IEEE Congress on Evolutionary Computation.

[42]  A. E. Eiben,et al.  From evolutionary computation to the evolution of things , 2015, Nature.

[43]  Dervis Karaboga,et al.  A powerful and efficient algorithm for numerical function optimization: artificial bee colony (ABC) algorithm , 2007, J. Glob. Optim..