Superpixel Benchmark and Comparison

Superpixel segmentation showed to be a useful preprocessing step in many computer vision applications. This led to a variety of algorithms to compute superpixel segmentations, each with individual strengths and weaknesses. We discuss the need for a standardized evaluation scheme of such algorithms and propose a benchmark including data sets, error metrics, usage guidelines and an open source implementation of the benchmark in a Matlab toolbox. The benchmark evaluates the quality of the superpixel segmentation with respect to human ground truth segmentation and the segmentation robustness to affine image transformations, which is crucial for application on image sequences. To our knowledge, this is the first benchmark considering the segmentation robustness to such image transformations. To better consider the characteristics of an image oversegmentation, we provide a new formulation of the undersegmentation error. Using this benchmark, we evaluate eight algorithms with available open source implementations and discuss the results with respect to requirements of further applications and runtime.