A study on the use of non-parametric tests for analyzing the evolutionary algorithms’ behaviour: a case study on the CEC’2005 Special Session on Real Parameter Optimization

In recent years, there has been a growing interest for the experimental analysis in the field of evolutionary algorithms. It is noticeable due to the existence of numerous papers which analyze and propose different types of problems, such as the basis for experimental comparisons of algorithms, proposals of different methodologies in comparison or proposals of use of different statistical techniques in algorithms’ comparison.In this paper, we focus our study on the use of statistical techniques in the analysis of evolutionary algorithms’ behaviour over optimization problems. A study about the required conditions for statistical analysis of the results is presented by using some models of evolutionary algorithms for real-coding optimization. This study is conducted in two ways: single-problem analysis and multiple-problem analysis. The results obtained state that a parametric statistical analysis could not be appropriate specially when we deal with multiple-problem results. In multiple-problem analysis, we propose the use of non-parametric statistical tests given that they are less restrictive than parametric ones and they can be used over small size samples of results. As a case study, we analyze the published results for the algorithms presented in the CEC’2005 Special Session on Real Parameter Optimization by using non-parametric test procedures.

[1]  Saku Kukkonen,et al.  Real-parameter optimization with differential evolution , 2005, 2005 IEEE Congress on Evolutionary Computation.

[2]  Kalyanmoy Deb,et al.  A population-based, steady-state procedure for real-parameter optimization , 2005, 2005 IEEE Congress on Evolutionary Computation.

[3]  Thomas Bartz-Beielstein,et al.  Experimental Research in Evolutionary Computation - The New Experimentalism , 2010, Natural Computing Series.

[4]  Irene A. Stegun,et al.  Handbook of Mathematical Functions. , 1966 .

[5]  Jing J. Liang,et al.  Dynamic multi-swarm particle swarm optimizer with local search , 2005, 2005 IEEE Congress on Evolutionary Computation.

[6]  A. Kai Qin,et al.  Self-adaptive differential evolution algorithm for numerical optimization , 2005, 2005 IEEE Congress on Evolutionary Computation.

[7]  David T. Morse,et al.  Minsize2: a Computer Program for Determining Effect Size and Minimum Sample Size for Statistical Significance for Univariate, Multivariate, and Nonparametric Tests , 1999 .

[8]  Ritu Gupta,et al.  Statistical exploratory analysis of genetic algorithms , 2004, IEEE Transactions on Evolutionary Computation.

[9]  Héctor Pomares,et al.  Statistical analysis of the main parameters involved in the design of a genetic algorithm , 2002, IEEE Trans. Syst. Man Cybern. Part C.

[10]  S. Holm.  A Simple Sequentially Rejective Multiple Test Procedure , 1979 .

[11]  Petr Posík,et al.  Real-parameter optimization using the mutation step co-evolution , 2005, 2005 IEEE Congress on Evolutionary Computation.

[12]  J. Shaffer.  Multiple Hypothesis Testing , 1995 .

[13]  Oliver Kramer,et al.  An experimental analysis of evolution strategies and particle swarm optimisers using design of experiments , 2007, GECCO '07.

[14]  Marcus Gallagher,et al.  On building a principled framework for evaluating and testing evolutionary algorithms: a continuous landscape generator , 2003, The 2003 Congress on Evolutionary Computation, 2003. CEC '03..

[15]  Francisco Herrera,et al.  Adaptive local search parameters for real-coded memetic algorithms , 2005, 2005 IEEE Congress on Evolutionary Computation.

[16]  Marcus Gallagher,et al.  Experimental results for the special session on real-parameter optimization at CEC 2005: a simple, continuous EDA , 2005, 2005 IEEE Congress on Evolutionary Computation.

[17]  C. Read,et al.  Handbook of the normal distribution , 1982 .

[18]  David H. Wolpert,et al.  No free lunch theorems for optimization , 1997, IEEE Trans. Evol. Comput..

[19]  John N. Hooker,et al.  Testing heuristics: We have it all wrong , 1995, J. Heuristics.

[20]  R. Iman,et al.  Approximations of the critical region of the fbietkan statistic , 1980 .

[21]  Marcus Gallagher,et al.  A general-purpose tunable landscape generator , 2006, IEEE Transactions on Evolutionary Computation.

[22]  Y. Hochberg.  A sharper Bonferroni procedure for multiple tests of significance , 1988 .

[23]  Babur Ozcelik,et al.  Comparison of the warpage optimization in the plastic injection molding using ANOVA, neural network model and genetic algorithm , 2006 .

[24]  M. Abramowitz,et al.  Handbook of Mathematical Functions With Formulas, Graphs and Mathematical Tables (National Bureau of Standards Applied Mathematics Series No. 55) , 1965 .

[25]  Anne Auger,et al.  Performance evaluation of an advanced local search evolutionary algorithm , 2005, 2005 IEEE Congress on Evolutionary Computation.

[26]  Carlos García-Martínez,et al.  Hybrid real-coded genetic algorithms with female and male differentiation , 2005, 2005 IEEE Congress on Evolutionary Computation.

[27]  Pedro J. Ballester,et al.  Real-parameter optimization performance study on the CEC-2005 benchmark with SPC-PNX , 2005, 2005 IEEE Congress on Evolutionary Computation.

[28]  Jing J. Liang,et al.  Problem Definitions and Evaluation Criteria for the CEC 2005 Special Session on Real-Parameter Optimization , 2005 .

[29]  César Hervás-Martínez,et al.  Improving crossover operator for real-coded genetic algorithms using virtual parents , 2007, J. Heuristics.

[30]  Janez Demsar,et al.  Statistical Comparisons of Classifiers over Multiple Data Sets , 2006, J. Mach. Learn. Res..

[31]  David J. Sheskin,et al.  Handbook of Parametric and Nonparametric Statistical Procedures , 1997 .

[32]  L. Darrell Whitley,et al.  Evaluating Evolutionary Algorithms , 1996, Artif. Intell..

[33]  G. E. Noether,et al.  Sample Size Determination for Some Common Nonparametric Tests , 1987 .

[34]  S. P. Wright,et al.  Adjusted P-values for simultaneous inference , 1992 .

[35]  L. Darrell Whitley,et al.  Test driving three 1995 genetic algorithms: New test functions and geometric matching , 1995, J. Heuristics.

[36]  Nikolaus Hansen,et al.  A restart CMA evolution strategy with increasing population size , 2005, 2005 IEEE Congress on Evolutionary Computation.