A Critical Review of "A Practical Guide to Select Quality Indicators for Assessing Pareto-Based Search Algorithms in Search-Based Software Engineering": Essay on Quality Indicator Selection for SBSE

This paper presents a critical review of the work published at ICSE'2016 on a practical guide of quality indicator selection for assessing multiobjective solution sets in search-based software engineering (SBSE). This review has two goals. First, we aim at explaining why we disagree with the work at ICSE'2016 and why the reasons behind this disagreement are important to the SBSE community. Second, we aim at providing a more clarified guide of quality indicator selection, serving as a new direction on this particular topic for the SBSE community. In particular, we argue that it does matter which quality indicator to select, whatever in the same quality category or across different categories. This claim is based upon the fundamental goal of multiobjective optimisation — supplying the decision-maker a set of solutions which are the most consistent with their preferences.

[1]  Anthony Ventresque,et al.  Is seeding a good strategy in multi-objective feature selection when feature models evolve? , 2017, Inf. Softw. Technol..

[2]  Hisao Ishibuchi,et al.  Modified Distance Calculation in Generational Distance and Inverted Generational Distance , 2015, EMO.

[3]  Robert M. Hierons,et al.  Multi-objective optimisation for regression testing , 2016, Inf. Sci..

[4]  Jyrki Wallenius,et al.  Quantitative Comparison of Approximate Solution Sets for Multicriteria Optimization Problems with Weighted Tchebycheff Preference Function , 2010, Oper. Res..

[5]  Qingfu Zhang,et al.  Combining Model-based and Genetics-based Offspring Generation for Multi-objective Optimization Using a Convergence Criterion , 2006, 2006 IEEE International Conference on Evolutionary Computation.

[6]  Lothar Thiele,et al.  Multiobjective evolutionary algorithms: a comparative case study and the strength Pareto approach , 1999, IEEE Trans. Evol. Comput..

[7]  Carlos A. Coello Coello,et al.  A Study of the Parallelization of a Coevolutionary Multi-objective Evolutionary Algorithm , 2004, MICAI.

[8]  Marc Roubens,et al.  Multiple criteria decision making , 1994 .

[9]  Marco Laumanns,et al.  Performance assessment of multiobjective optimizers: an analysis and review , 2003, IEEE Trans. Evol. Comput..

[10]  Yves Le Traon,et al.  Combining Multi-Objective Search and Constraint Solving for Configuring Large Software Product Lines , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[11]  Rami Bahsoon,et al.  Self-Adaptive Trade-off Decision Making for Autoscaling Cloud-Based Services , 2016, IEEE Transactions on Services Computing.

[12]  Sergio Segura,et al.  SIP: Optimal Product Selection from Feature Models Using Many-Objective Evolutionary Optimization , 2016, ACM Trans. Softw. Eng. Methodol..

[13]  Hiroshi Wada,et al.  E³: A Multiobjective Optimization Framework for SLA-Aware Service Composition , 2012, IEEE Transactions on Services Computing.

[14]  Lothar Thiele,et al.  The Hypervolume Indicator Revisited: On the Design of Pareto-compliant Indicators Via Weighted Integration , 2007, EMO.

[15]  David A. Van Veldhuizen,et al.  Evolutionary Computation and Convergence to a Pareto Front , 1998 .

[16]  Serpil Sayin,et al.  Measuring the quality of discrete representations of efficient sets in multiple objective mathematical programming , 2000, Math. Program..

[17]  Liam Murphy,et al.  VM reassignment in hybrid clouds for large decentralised companies: A multi-objective challenge , 2018, Future Gener. Comput. Syst..

[18]  Martin Monperrus,et al.  A critical review of "automatic patch generation learned from human-written patches": essay on the problem statement and the evaluation of automatic software repair , 2014, ICSE.

[19]  Yan Li,et al.  A Practical Guide to Select Quality Indicators for Assessing Pareto-Based Search Algorithms in Search-Based Software Engineering , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE).

[20]  Shengxiang Yang,et al.  Diversity Comparison of Pareto Front Approximations in Many-Objective Optimization , 2014, IEEE Transactions on Cybernetics.