Uncertainty based model selection for fast semantic segmentation

Semantic segmentation approaches can largely be divided into two categories. One with accurate results but slow inference, and another one with real-time inference but sacrificing some performance for speed. In this paper, we try to exploit the benefits of both categories, i.e. accuracy and speed, through the use of model selection techniques. Using the uncertainty, calculated from the entropy map, as our selection criterion, we leverage the speed of the fast, but not so accurate, model for regions with high certainty, that comprise the majority of the input image, while for a few, carefully selected regions with low certainty we employ an accurate, yet expensive, model, to predict the semantic labels. Our experimental results show that our method greatly boosts the performance of the baseline model, while retaining reasonable inference speeds.