Comparing multi-objective optimization algorithms using an ensemble of quality indicators with deep statistical comparison approach

This paper presents a study on making a statistical comparison of multi-objective optimization algorithms using an ensemble of quality indicators together with a deep statistical comparison (DSC) approach. The DSC approach has been recently proposed for statistically comparing meta-heuristic stochastic optimization algorithms for single-objective problems. The DSC ranking scheme is based on the whole distribution, rather than on one statistic such as either the average or the median. This study uses two ensemble combiners to rank and compare algorithms using the DSC ranking scheme for each quality indicator for a given problem. Experimental results performed using 3 multi-objective optimization algorithms on 16 test problems show that ensembles of quality indicators with transformed DSC rankings give more robust results than when the same ensembles are used with transformed rankings obtained by some standard ranking schemes.

[1]  Benjamín Barán,et al.  Performance metrics in multi-objective optimization , 2015, 2015 Latin American Computing Conference (CLEI).

[2]  Tome Eftimov,et al.  DISADVANTAGES OF STATISTICAL COMPARISON OF STOCHASTIC OPTIMIZATION ALGORITHMS , 2016 .

[3]  Francisco Herrera,et al.  A study on the use of non-parametric tests for analyzing the evolutionary algorithms’ behaviour: a case study on the CEC’2005 Special Session on Real Parameter Optimization , 2009, J. Heuristics.

[4]  Carlos M. Fonseca,et al.  Inferential Performance Assessment of Stochastic Optimisers and the Attainment Function , 2001, EMO.

[5]  Gary B. Lamont,et al.  Evolutionary Algorithms for Solving Multi-Objective Problems , 2002, Genetic Algorithms and Evolutionary Computation.

[6]  P. N. Suganthan,et al.  Ensemble of niching algorithms , 2010, Inf. Sci..

[7]  Tea Tusar,et al.  Differential Evolution versus Genetic Algorithms in Multiobjective Optimization , 2007, EMO.

[8]  Enrique Alba,et al.  The jMetal framework for multi-objective optimization: Design and architecture , 2010, IEEE Congress on Evolutionary Computation.

[9]  Tome Eftimov,et al.  A Novel Approach to statistical comparison of meta-heuristic stochastic optimization algorithms using deep statistics , 2017, Inf. Sci..

[10]  Sandrine Dudoit,et al.  Multiple Testing. Part II. Step-Down Procedures for Control of the Family-Wise Error Rate , 2004, Statistical applications in genetics and molecular biology.

[11]  A. Shamsai,et al.  Multi-objective Optimization , 2017, Encyclopedia of Machine Learning and Data Mining.

[12]  R. Lyndon While,et al.  A Scalable Multi-objective Test Problem Toolkit , 2005, EMO.

[13]  Thomas G. Dietterich.  Multiple Classifier Systems , 2000, Lecture Notes in Computer Science.

[14]  Marco Laumanns,et al.  Scalable Test Problems for Evolutionary Multiobjective Optimization , 2005, Evolutionary Multiobjective Optimization.