How to Save your Annotation Cost for Panoptic Segmentation?

How to properly reduce the annotation cost for panoptic segmentation? How to leverage and optimize the cost-quality trade-off for training data and model? These questions are key challenges towards a label-efficient and scalable panoptic segmentation system due to its expensive instance/semantic pixel-level annotation requirements. By closely examining different kinds of cheaper labels, we introduce a novel multiobjective framework to automatically determine the allocation of different annotations, so as to reach a better segmentation quality with a lower annotation cost. Specifically, we design a Cost-Quality Balanced Network (CQB-Net) to generate the panoptic segmentation map, which distills the crucial relations between various supervisions including panoptic labels, image-level classification labels, bounding boxes, and the semantic coherence information between the foreground and background. Instead of ad-hoc allocation during training, we formulate the optimization of cost-quality trade-off as a Multi-Objective Optimization Problem (MOOP). We model the marginal quality improvement of each annotation and approximate the Pareto-front to enable a label-efficient allocation ratio. Extensive experiments on COCO benchmark show the superiority of our method, e.g. achieving a segmentation quality of 43.4% compared to 43.0% of OCFusion while saving 2.4x annotation cost. Introduction Panoptic segmentation unifies foreground instance segmentation (named thing) and semantic segmentation on amorphous background regions (named stuff ) to generate rich and coherent masks. It poses a challenging problem on holistic image understanding of all foreground objects and background contents simultaneously. State-of-the-art methods (Yang et al. 2019b) are trained in a fully-supervised fashion, for which per-pixel background class and foreground instance labels are required. Due to the data-hungry nature of deep networks, these approaches demand an enormous number of training images with curated groundtruth labels, which are given by hand in general. However, manual annotation of such labels is extremely labor-intensive and time-consuming, e.g. taking around 19 minutes for one single image in COCO to be fully annotated (Caesar, Uijlings, ∗Corresponding Author Copyright c © 2021, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. O1 O2 O3 O4 O5 O6 R1 R2 R3 R4 R5 R6 R7