Propose and Re-rank Semantic Segmentation via Deep Image Classification

Image classification (i.e. the task of predicting the presence or absence of an object category in an image) is the poor man’s first step towards automatic image understanding. Object detection (i.e. localizing the object in the image) and semantic image segmentation (i.e. delineating the boundaries of an object) are progressively more detailed and arguably more useful image understanding tasks. Recently, the development of large-scale datasets such as ImageNet [5] has enabled a resurgence of interest in image classification [8, 12–15, 18] especially in the context of learning representations [2] via deep convolutional neural networks (CNN) [6,7,10]. At ILSVRC 2012, Krizhevsky et al. [10] presented a convolutional neural network (CNN) that significantly outperformed other entries in the image classification task, which was later vindicated by repeat performance at ILSVRC 2013. These improvements in image classification have been shown to generalize well and even help more detailed vision tasks. Donahue [6] showed that intermediate responses of these CNNs can be used as features that generalize well to classification tasks on other datasets. Girshick et al. [7] showed that classification CNNs can also be adapted to perform state-of-art object detection. In this work, we study how improvements in image classification can benefit semantic segmentation. Specifically, we study a “propose and re-rank” framework that has gained popularity in the semantic segmentation literature, and has been shown to achieve state-of-theart results on the PASCAL segmentation challenge [3, 17]. Fig. 1 shows an illustration. Diverse multiple proposals of either regions or entire segmentation maps are produced and then re-ranked so that better proposals score higher. The proposals are not perfect and this results in propagation of errors along the pipeline. The benefit of this approach however is that the re-ranking problem is simple – it involves choosing 1 from just M proposals. It is thus amenable to global optimization. It allows for incorporation of arbitrary features and high-order interactions. One can thus conveniently “plug-and-play” image classification as a “submodule” in the re-ranking step for semantic segmentation. Image classifier Classification PASCAL mean AP(%) accuracy(%) Random 7.41 44.81 Gist + linear SVM 26.83 43.89 SUN [16] kernels + SVM 60.60 44.90 DeCAF7 [6] + linear SVM + dropout 74.52 45.61 DeCAF7 + RBF SVM 76.85 46.93 Ground truth 100.00 53.22 Table 1. Image classification performance versus semantic segmentation performance when re-ranking with the corresponding image classification feature alone on PASCAL VOC12 val.