A statistical methodology for analysing heuristic algorithms

Heuristic experimentation commonly entails running an algorithm on the instances of some standard benchmark problem set and measuring its performance solution quality, run time or both. These performance results are then compared with the results other heuristic algorithms obtained on this benchmark problem set. It is a type of evaluation that ensues a competition with state-ofthe-art methods in the literature. The goal is to obtain a better solution quality and/or a faster running time on the benchmark instances than other existing algorithm and claim rst place in the horse race . This approach, however, does not seek to explain why one method performs better than another one [2]. Which elements of the heuristic algorithm have contriubted to a greater or less extent to this superior performance ? Is it mainly due to a certain (combination of) operator(s) employed within the algorithm? Or xing certain parameters at speci c values ? Or maybe it is due to a researcher's superior coding skills leading to a more e cient implementation of an existing algorithm? Do all components signicantly contribute to the performance of the algorithm, or can certain elements be left out, thereby possibly increasing the e ciency of the method ? These are all questions that often remain unanswered when a new method is presented. Even though some competition between researchers might spur innovation, it has been noted that true innovation builds on the understanding of how a heuristic algorithm behaves, and not on proof of competitiveness [5]. A competitive focus works when considering a speci c setting [4], but when the objective is to learn how the di erent heuristic elements contribute to performance and make statements beyond a speci c problem setting, a statistical evaluation methodology has to be applied.