Panoptic Segmentation Meets Remote Sensing

Deep Learning (DL) methods achieved state-of-the-art results in remote sensing image segmentation studies with an increasing trend. Most studies focus on semantic and instance segmentation methods, with a research gap in panoptic segmentation. Panoptic segmentation combines instance and semantic predictions, allowing the detection of ”things” (countable objects) and ”stuff” (different backgrounds) simultaneously. Effectively approaching panoptic segmentation in remotely sensed data can be auspicious in many challenging problems since it allows continuous mapping and specific target counting. Several difficulties have prevented the growth of this task in remote sensing: (a) most algorithms are designed for traditional images, (b) image labeling must encompass ”things” and ”stuff” classes (being much more laborious), and (c) the annotation format is complex. Thus, aiming to solve and increase the operability of panoptic segmentation in remote sensing, this study has five objectives: (1) create a novel data preparation pipeline for the panoptic segmentation task using GIS tools, (2) propose a novel annotation conversion software to generate panoptic annotations in the COCO format automatically; (3) propose a novel dataset on urban areas, (4) modify and leverage the Detectron2 architecture and software for the task, and (5) evaluate semantic, instance, and panoptic metrics and present the difficulties of this task in the urban setting. We used an aerial image with a 0,24-meter spatial resolution in the city of Braśılia, covering an area of 79,401 m. The annotations considered fourteen classes (three ”stuff” and eleven ”thing” categories). Our proposed pipeline considers three image inputs (original image, semantic image, and panoptic image). The proposed software uses these inputs ∗Corresponding author: osmarjr@unb.br Email addresses: osmarcarvalho@ieee.org (Osmar Luiz Ferreira de Carvalho), cristiano@dubbox.org (Cristiano Rosa e Silva), anesmar@ieee.org (Anesmar Olino de Albuquerque), nickolas.santana@unb.br (Nickolas Castro Santana), dibio@unb.br (Dibio Leandro Borges), robertogomes@unb.br (Roberto Arnaldo Trancoso Gomes), renatofg@unb.br (Renato Fontes Guimarães) Preprint December 1, 2021 ar X iv :2 11 1. 12 12 6v 2 [ cs .C V ] 3 0 N ov 2 02 1 alongside point shapefiles, creating samples at the centroid of each point shapefile with their corresponding annotations in the COCO format. The usage of points allows the researchers to choose samples in critical areas. Our study generated 3,400 samples with 512x512 pixel dimensions (3,000 for training, 200 for validation, and 200 for testing). The analysis used the Panoptic-FPN model with two backbones (ResNet-50 and ResNet-101), and the model evaluation considered three metric types (semantic metrics, instance metrics, and panoptic metrics). Regarding the main metrics, we obtained 93.865, 47.691, and 64.979 for the mean Intersection over Union, box Average Precision, and Panoptic Quality, respectively. Our study presents the first effective pipeline for panoptic segmentation and an extensive database for other researchers to use and deal with other data or related problems requiring a thorough scene understanding.