Performance metrics in multi-objective optimization

In the last decades, a large number of metrics has been proposed to compare the performance of different evolutionary approaches in multi-objective optimization. This situation leads to difficulties when comparisons among the output of different algorithms are needed and appropriate metrics must be selected to perform those comparisons. Hence, no complete agreement on what metrics should be used exists. This paper presents a review and analysis of 54 multi-objective-optimization metrics in the specialized literature, discussing the usage, tendency and advantages/disadvantages of the most cited ones in order to give researchers enough information when choosing metrics is necessary. The review process performed in this work indicates that the hypervolume is the most used metric, followed by the generational distance, the epsilon indicator and the inverted generational distance.

[1]  Carlos A. Coello Coello,et al.  Improving PSO-Based Multi-objective Optimization Using Crowding, Mutation and epsilon-Dominance , 2005, EMO.

[2]  Hajime Kita,et al.  Individual Evaluation Scheduling for Experiment-Based Evolutionary Multi-objective Optimization , 2007, EMO.

[3]  Jason R. Schott.  Fault Tolerant Design Using Single and Multicriteria Genetic Algorithm Optimization. , 1995 .

[4]  David W. Corne,et al.  Approximating the Nondominated Front Using the Pareto Archived Evolution Strategy , 2000, Evolutionary Computation.

[5]  Matteo Nicolini,et al.  EVALUATING PERFORMANCE OF MULTI-OBJECTIVE GENETIC ALGORITHMS FOR WATER DISTRIBUTION SYSTEM OPTIMIZATION , 2004 .

[6]  Gary B. Lamont,et al.  Evolutionary Algorithms for Solving Multi-Objective Problems , 2002, Genetic Algorithms and Evolutionary Computation.

[7]  David Greiner,et al.  Gray Coding in Evolutionary Multicriteria Optimization: Application in Frame Structural Optimum Design , 2005, EMO.

[8]  Nicola Beume,et al.  On the Complexity of Computing the Hypervolume Indicator , 2009, IEEE Transactions on Evolutionary Computation.

[9]  Everardo Gutiérrez,et al.  Multi-objective Go with the Winners Algorithm: A Preliminary Study , 2005, EMO.

[10]  Kalyanmoy Deb,et al.  A fast and elitist multiobjective genetic algorithm: NSGA-II , 2002, IEEE Trans. Evol. Comput..

[11]  Kalyanmoy Deb,et al.  Running performance metrics for evolutionary multi-objective optimizations , 2002 .

[12]  Lothar Thiele,et al.  Multiobjective Optimization Using Evolutionary Algorithms - A Comparative Case Study , 1998, PPSN.

[13]  Douglas B. Kell,et al.  Computational cluster validation in post-genomic data analysis , 2005, Bioinform..

[14]  Carlos M. Fonseca,et al.  Exploring the Performance of Stochastic Multiobjective Optimisers with the Second-Order Attainment Function , 2005, EMO.

[15]  Tomoyuki Hiroyasu,et al.  DIVIDED RANGE GENETIC ALGORITHMS IN MULTIOBJECTIVE OPTIMIZATION PROBLEMS , 1999 .

[16]  Carlos A. Coello Coello,et al.  Solving Multiobjective Optimization Problems Using an Artificial Immune System , 2005, Genetic Programming and Evolvable Machines.

[17]  Ching-Lai Hwang,et al.  Multiple attribute decision making : an introduction , 1995 .

[18]  Yaochu Jin,et al.  A Critical Survey of Performance Indices for Multi-Objective Optimisation , 2003 .

[19]  Qingfu Zhang,et al.  Combining Model-based and Genetics-based Offspring Generation for Multi-objective Optimization Using a Convergence Criterion , 2006, 2006 IEEE International Conference on Evolutionary Computation.

[20]  Heike Trautmann,et al.  A Taxonomy of Online Stopping Criteria for Multi-Objective Evolutionary Algorithms , 2011, EMO.

[21]  Hisao Ishibuchi,et al.  Mating Scheme for Controlling the Diversity-Convergence Balance for Multiobjective Optimization , 2004, GECCO.

[22]  Heike Trautmann,et al.  On the properties of the R2 indicator , 2012, GECCO '12.

[23]  Marc Schoenauer,et al.  A Steady Performance Stopping Criterion for Pareto-based Evolutionary Algorithms , 2004 .

[24]  Hisao Ishibuchi,et al.  Evolutionary many-objective optimization , 2008, 2008 3rd International Workshop on Genetic and Evolving Systems.

[25]  Peter Vamplew,et al.  The Combative Accretion Model - Multiobjective Optimisation Without Explicit Pareto Ranking , 2005, EMO.

[26]  大林 茂.  Evolutionary Multi-Criterion Optimization : 4th International Conference, EMO 2007, Matsushima, Japan, March 5-8, 2007 : proceedings , 2007 .

[27]  Robert G. Reynolds,et al.  Memorial tribute to Dr. Michael Conrad , 2001, IEEE Trans. Evol. Comput..

[28]  Ofer M. Shir,et al.  Enhancing Decision Space Diversity in Evolutionary Multiobjective Algorithms , 2009, EMO.

[29]  Hisao Ishibuchi,et al.  Evolutionary many-objective optimization: A short review , 2008, 2008 IEEE Congress on Evolutionary Computation (IEEE World Congress on Computational Intelligence).

[30]  Dirk Thierens,et al.  The Naive MIDEA: A Baseline Multi-objective EA , 2005, EMO.

[31]  Lothar Thiele,et al.  The Hypervolume Indicator Revisited: On the Design of Pareto-compliant Indicators Via Weighted Integration , 2007, EMO.

[32]  Hisao Ishibuchi,et al.  Difficulties in specifying reference points to calculate the inverted generational distance for many-objective optimization problems , 2014, 2014 IEEE Symposium on Computational Intelligence in Multi-Criteria Decision-Making (MCDM).

[33]  Michael T. M. Emmerich,et al.  Test Problems Based on Lamé Superspheres , 2007, EMO.

[34]  Salvatore Greco,et al.  Evolutionary Multi-Criterion Optimization , 2011, Lecture Notes in Computer Science.

[35]  Hisao Ishibuchi,et al.  A multi-objective genetic local search algorithm and its application to flowshop scheduling , 1998, IEEE Trans. Syst. Man Cybern. Part C.

[36]  Lothar Thiele,et al.  Multiobjective evolutionary algorithms: a comparative case study and the strength Pareto approach , 1999, IEEE Trans. Evol. Comput..

[37]  Qingfu Zhang,et al.  This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination. IEEE TRANSACTIONS ON EVOLUTIONARY COMPUTATION 1 RM-MEDA: A Regularity Model-Based Multiobjective Estimation of , 2022 .

[38]  R. K. Ursem.  Multi-objective Optimization using Evolutionary Algorithms , 2009 .

[39]  Gary B. Lamont,et al.  Multiobjective evolutionary algorithms: classifications, analyses, and new innovations , 1999 .

[40]  Timothy M. Chan.  Klee's Measure Problem Made Easy , 2013, 2013 IEEE 54th Annual Symposium on Foundations of Computer Science.

[41]  C. Coello,et al.  Improving PSO-based Multi-Objective Optimization using Crowding , Mutation and �-Dominance , 2005 .

[42]  Shapour Azarm,et al.  Metrics for Quality Assessment of a Multiobjective Design Optimization Solution Set , 2001 .

[43]  Ricardo H. C. Takahashi.  Evolutionary multi-criterion optimization : 6th international conference, EMO 2011, Ouro Preto, Brazil, April 5-8, 2011 : proceedings , 2011 .

[44]  Patrick M. Reed,et al.  The Value of Online Adaptive Search: A Performance Comparison of NSGAII, epsilon-NSGAII and epsilonMOEA , 2005, EMO.

[45]  Jonathan E. Fieldsend,et al.  Using unconstrained elite archives for multiobjective optimization , 2003, IEEE Trans. Evol. Comput..

[46]  Marco Laumanns,et al.  Performance assessment of multiobjective optimizers: an analysis and review , 2003, IEEE Trans. Evol. Comput..

[47]  Kay Chen Tan,et al.  An Investigation on Noisy Environments in Evolutionary Multiobjective Optimization , 2007, IEEE Transactions on Evolutionary Computation.

[48]  Carlos M. Fonseca,et al.  An Improved Dimension-Sweep Algorithm for the Hypervolume Indicator , 2006, 2006 IEEE International Conference on Evolutionary Computation.

[49]  Jesús García,et al.  A cumulative evidential stopping criterion for multiobjective optimization evolutionary algorithms , 2007, GECCO '07.

[50]  El-Ghazali Talbi,et al.  A multiobjective genetic algorithm for radio network optimization , 2000, Proceedings of the 2000 Congress on Evolutionary Computation. CEC00 (Cat. No.00TH8512).

[51]  Carlos M. Fonseca,et al.  Evolutionary multi-criterion optimization : 7th international conference, EMO 2013, Sheffield, UK, March 19-22, 2013 : proceedings , 2013 .

[52]  Carlos A. Brizuela,et al.  Clustering Based Parallel Many-Objective Evolutionary Algorithms Using the Shape of the Objective Vectors , 2015, EMO.

[53]  Piotr Czyzżak,et al.  Pareto simulated annealing—a metaheuristic technique for multiple‐objective combinatorial optimization , 1998 .

[54]  Eckart Zitzler,et al.  Evolutionary algorithms for multiobjective optimization: methods and applications , 1999 .

[55]  Hamidreza Eskandari,et al.  FastPGA: A Dynamic Population Sizing Approach for Solving Expensive Multiobjective Optimization Problems , 2006, EMO.

[56]  Lothar Thiele,et al.  Comparison of Multiobjective Evolutionary Algorithms: Empirical Results , 2000, Evolutionary Computation.

[57]  Hussein A. Abbass,et al.  A dominance-based stability measure for multi-objective evolutionary algorithms , 2009, 2009 IEEE Congress on Evolutionary Computation.

[58]  Eckart Zitzler,et al.  Evolutionary Multi-Criterion Optimization, Third International Conference, EMO 2005, Guanajuato, Mexico, March 9-11, 2005, Proceedings , 2005, EMO.

[59]  Joshua D. Knowles,et al.  On metrics for comparing nondominated sets , 2002, Proceedings of the 2002 Congress on Evolutionary Computation. CEC'02 (Cat. No.02TH8600).

[60]  Jin-Kao Hao,et al.  Evolutionary Multi-Criterion Optimization, 5th International Conference, EMO 2009, Nantes, France, April 7-10, 2009. Proceedings , 2009, EMO.

[61]  Jinhua Zheng,et al.  Spread Assessment for Evolutionary Multi-Objective Optimization , 2009, EMO.

[62]  N. Stander,et al.  A non‐dominance‐based online stopping criterion for multi‐objective evolutionary algorithms , 2010 .

[63]  A. Farhang-Mehr,et al.  Diversity assessment of Pareto optimal solution sets: an entropy approach , 2002, Proceedings of the 2002 Congress on Evolutionary Computation. CEC'02 (Cat. No.02TH8600).

[64]  M. Hansen,et al.  Evaluating the quality of approximations to the non-dominated set , 1998 .