Ranking empirical cumulative distribution functions using stochastic and pareto dominance

In this paper, we propose two new approaches to rank the frequently used empirical cumulative distribution functions (ECDFs) for performance assessment of stochastic optimization algorithms. In the first approach, the different orders of stochastic dominance among running lengths are adopted in a hierarchical manner: the first order stochastic dominance is tested and the second order is used when the first order leads to incomparable results. In the second approach, ECDFs are considered as local Pareto front of the bi-criteria decision-making problem, in which one objective is to achieve a high success rate and the other is to use as few function evaluations as possible. In this case, it is proposed to adopt the multi-objective performance indicator to handle incomparable ECDFs.