Domain Adaptive Semantic Segmentation via Regional Contrastive Consistency Regularization

Unsupervised domain adaptation (UDA) for semantic seg-mentation has been well-studied in recent years. However, most existing works largely neglect the local regional consis-tency across different domains, and are less robust to changes in outdoor environments. In this paper, we propose a novel and fully end-to-end trainable approach, called regional contrastive consistency regularization (RCCR) for domain adaptive semantic segmentation. Our core idea is to pull the sim-ilar regional features extracted from the same location of dif-ferent images, i.e., the original image and augmented image, to be closer, and meanwhile push the features from the dif-ferent locations of the two images to be separated. We pro-pose a region-wise contrastive loss with two sampling strate-gies to realize effective regional consistency. Besides, we present momentum projection heads, where the teacher pro-jection head is the exponential moving average of the student. Finally, a memory bank mechanism is designed to learn more robust and stable region-wise features under varying environ-ments. Extensive experiments demonstrate that our approach outperforms the state-of-the-art methods.

[1]  Chase Rainwater,et al.  BiMaL: Bijective Maximum Likelihood Approach to Domain Adaptation in Semantic Scene Segmentation , 2021, IEEE International Conference on Computer Vision.

[2]  Jiaya Jia,et al.  Semi-supervised Semantic Segmentation with Directional Context-aware Consistency , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Fei Yu,et al.  DAST: Unsupervised Domain Adaptation in Semantic Segmentation Based on Discriminator Attention and Self-Training , 2021, AAAI.

[4]  Pascal Fua,et al.  Domain Adaptation for Semantic Segmentation via Patch-Wise Contrastive Learning , 2021, ArXiv.

[5]  Liang Zheng,et al.  Category-Level Adversarial Adaptation for Semantic Segmentation Using Purified Features , 2021, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[6]  L. Gool,et al.  Exploring Cross-Image Pixel Contrast for Semantic Segmentation , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[7]  Xiang Bai,et al.  Affinity Space Adaptation for Semantic Segmentation Across Domains , 2020, IEEE Transactions on Image Processing.

[8]  Yunchao Wei,et al.  Content-Consistent Matching for Domain Adaptive Semantic Segmentation , 2020, ECCV.

[9]  Amit K. Roy-Chowdhury,et al.  Domain Adaptive Semantic Segmentation Using Weak Labels , 2020, ECCV.

[10]  L. Svensson,et al.  DACS: Domain Adaptation via Cross-domain Mixed Sampling , 2020, 2021 IEEE Winter Conference on Applications of Computer Vision (WACV).

[11]  Wei Zhang,et al.  Classes Matter: A Fine-grained Adversarial Approach to Cross-domain Semantic Segmentation , 2020, ECCV.

[12]  Xiaobing Zhang,et al.  Contextual-Relation Consistent Domain Adaptation for Semantic Segmentation , 2020, ECCV.

[13]  In So Kweon,et al.  Unsupervised Intra-Domain Adaptation for Semantic Segmentation Through Self-Supervision , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Stefano Soatto,et al.  FDA: Fourier Domain Adaptation for Semantic Segmentation , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Wen-mei W. Hwu,et al.  Differential Treatment for Stuff and Things: A Simple Unsupervised Domain Adaptation Method for Semantic Segmentation , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Junzhou Huang,et al.  Label-Driven Reconstruction for Domain Adaptation in Semantic Segmentation , 2020, ECCV.

[17]  Hyeran Byun,et al.  Learning Texture Invariant Representation for Domain Adaptation of Semantic Segmentation , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Changick Kim,et al.  Self-Ensembling With GAN-Based Data Augmentation for Domain Adaptation in Semantic Segmentation , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[19]  Xiaofeng Liu,et al.  Confidence Regularized Self-Training , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[20]  Bo Du,et al.  Self-Ensembling Attention Networks: Addressing Domain Shift for Semantic Segmentation , 2019, AAAI.

[21]  Timo Aila,et al.  Semi-supervised semantic segmentation needs strong, varied perturbations , 2019, BMVC.

[22]  Nuno Vasconcelos,et al.  Bidirectional Learning for Domain Adaptation of Semantic Segmentation , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  B. V. Vijaya Kumar,et al.  Unsupervised Domain Adaptation for Semantic Segmentation via Class-Balanced Self-training , 2018, ECCV.

[24]  Ming-Hsuan Yang,et al.  Learning to Adapt Structured Output Space for Semantic Segmentation , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[25]  Vladlen Koltun,et al.  Playing for Data: Ground Truth from Computer Games , 2016, ECCV.

[26]  Antonio M. López,et al.  The SYNTHIA Dataset: A Large Collection of Synthetic Images for Semantic Segmentation of Urban Scenes , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  Iasonas Kokkinos,et al.  DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[28]  Sebastian Ramos,et al.  The Cityscapes Dataset for Semantic Urban Scene Understanding , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  Li Yunsheng,et al.  Bidirectional Learning for Domain Adaptation of Semantic Segmentation , 2019 .