Semantic Distribution-aware Contrastive Adaptation for Semantic Segmentation

Domain adaptive semantic segmentation refers to making accurate dense predictions on a certain target domain with only pixel-level annotations of a specific source domain. Current state-of-the-art works suggest that performing alignment from the category perspective can alleviate domain shift reasonably. However, they are mainly based on image-to-image adversarial training and little consideration is given to semantic variations of an object among different images. A possible consequence is that such alignment fails to capture a comprehensive picture of different categories, leading to unstable category alignment and limited generalization advances. This motivates us to explore a holistic representative, the semantic distribution from each category in the source domain, to mitigate the problem above. In this paper, we present a new semantic distribution-aware contrastive adaptation algorithm, dubbed as SDCA, that enables pixel-wise representation alignment across domains under the guidance of the semantic distributions. To be precise, we first design a novel contrastive loss at pixel level by considering the correspondences between the semantic distributions and pixel-wise representations from both domains. Essentially, clusters of pixel representations from the same category are obliged to cluster together and those from different categories are obliged to spread out, boosting segmentation capability of the model. Next, an upper bound on this formulation is derived by implicitly involving the simultaneous learning of an infinite number of (dis)similar pixel pairs, making it highly efficient. Though simple, we empirically unveil the certain mechanisms that promote the potential of SDCA. Finally, we verify that SDCA can further improve the segmentation accuracy when integrated with the self-supervised learning method. We evaluate the proposed method on multiple benchmarks, achieving considerable improvements over existing algorithms.