Parameterization of state-of-the-art performance indicators: a robustness study based on inexact TSP solvers

Performance comparisons of optimization algorithms are heavily influenced by the underlying indicator(s). In this paper we investigate commonly used performance indicators for single-objective stochastic solvers, such as the Penalized Average Runtime (e.g., PAR10) or the Expected Running Time (ERT), based on exemplary benchmark performances of state-of-the-art inexact TSP solvers. Thereby, we introduce a methodology for analyzing the effects of (usually heuristically set) indicator parametrizations - such as the penalty factor and the method used for aggregating across multiple runs - w.r.t. the robustness of the considered optimization algorithms.

[1]  A. E. Eiben,et al.  Introduction to Evolutionary Computing , 2003, Natural Computing Series.

[2]  L. Darrell Whitley,et al.  Building a better heuristic for the traveling salesman problem: combining edge assembly crossover and partition crossover , 2017, GECCO.

[3]  David H. Wolpert,et al.  No free lunch theorems for optimization , 1997, IEEE Trans. Evol. Comput..

[4]  R. Geoff Dromey,et al.  An algorithm for the selection problem , 1986, Softw. Pract. Exp..

[5]  Shigenobu Kobayashi,et al.  A Powerful Genetic Algorithm Using Edge Assembly Crossover for the Traveling Salesman Problem , 2013, INFORMS J. Comput..

[6]  Bernd Bischl,et al.  ASlib: A benchmark library for algorithm selection , 2015, Artif. Intell..

[7]  Keld Helsgaun,et al.  General k-opt submoves for the Lin–Kernighan TSP heuristic , 2009, Math. Program. Comput..

[8]  Heike Trautmann,et al.  Leveraging TSP Solver Complementarity through Machine Learning , 2018, Evolutionary Computation.

[9]  Thomas Stützle,et al.  On the Empirical Scaling Behaviour of State-of-the-art Local Search Algorithms for the Euclidean TSP , 2015, GECCO.

[10]  Richard M. Karp,et al.  Reducibility Among Combinatorial Problems , 1972, 50 Years of Integer Programming.

[11]  Heike Trautmann,et al.  Multi-objective Performance Measurement: Alternatives to PAR10 and Expected Running Time , 2018, LION.

[12]  Raymond Ros,et al.  Real-Parameter Black-Box Optimization Benchmarking 2009: Experimental Setup , 2009 .

[13]  Heike Trautmann,et al.  Improving the State of the Art in Inexact TSP Solving Using Per-Instance Algorithm Selection , 2015, LION.

[14]  Lars Kotthoff,et al.  Algorithm Selection for Combinatorial Search Problems: A Survey , 2012, AI Mag..

[15]  Jiming Liu,et al.  Multiagent Optimization System for Solving the Traveling Salesman Problem (TSP) , 2009, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).