DAugNet: Unsupervised, Multisource, Multitarget, and Life-Long Domain Adaptation for Semantic Segmentation of Satellite Images

The domain adaptation of satellite images has recently gained increasing attention to overcome the limited generalization abilities of machine learning models when segmenting large-scale satellite images. Most of the existing approaches seek for adapting the model from one domain to another. However, such single-source and single-target setting prevents the methods from being scalable solutions since, nowadays, multiple sources and target domains having different data distributions are usually available. Besides, the continuous proliferation of satellite images necessitates the classifiers to adapt to continuously increasing data. We propose a novel approach, coined DAugNet, for unsupervised, multisource, multitarget, and life-long domain adaptation of satellite images. It consists of a classifier and a data augmentor. The data augmentor, which is a shallow network, is able to perform style transfer between multiple satellite images in an unsupervised manner, even when new data are added over time. In each training iteration, it provides the classifier with diversified data, which makes the classifier robust to large data distribution difference between the domains. Our extensive experiments prove that DAugNet significantly better generalizes to new geographic locations than the existing approaches.

[1]  Kurt Keutzer,et al.  Multi-source Domain Adaptation for Semantic Segmentation , 2019, NeurIPS.

[2]  Rafael C. González,et al.  Digital image processing, 3rd Edition , 2008 .

[3]  Liang Chen,et al.  A Fully Convolutional Tri-Branch Network (FCTN) for Domain Adaptation , 2018, 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[4]  Robert M. Haralick,et al.  Feature normalization and likelihood-based similarity measures for image retrieval , 2001, Pattern Recognit. Lett..

[5]  Alexandr A. Kalinin,et al.  Albumentations: fast and flexible image augmentations , 2018, Inf..

[6]  Raquel Urtasun,et al.  DeepRoadMapper: Extracting Road Topology from Aerial Images , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[7]  Dalton Lunga,et al.  Large Scale Unsupervised Domain Adaptation of Segmentation Networks with Adversarial Learning , 2019, IGARSS 2019 - 2019 IEEE International Geoscience and Remote Sensing Symposium.

[8]  Qi-Xing Huang,et al.  Domain Transfer Through Deep Activation Matching , 2018, ECCV.

[9]  Mikhail F. Kanevski,et al.  SVM-Based Boosting of Active Learning Strategies for Efficient Domain Adaptation , 2012, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[10]  Tengyu Ma,et al.  Understanding Self-Training for Gradual Domain Adaptation , 2020, ICML.

[11]  Lorenzo Bruzzone,et al.  Tensor Alignment Based Domain Adaptation for Hyperspectral Image Classification , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[12]  Pierre Alliez,et al.  Incremental Learning for Semantic Segmentation of Large-Scale Remote Sensing Data , 2018, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[13]  Pierre Alliez,et al.  Can semantic labeling methods generalize to any city? the inria aerial image labeling benchmark , 2017, 2017 IEEE International Geoscience and Remote Sensing Symposium (IGARSS).

[14]  Liang Lin,et al.  Deep Cocktail Network: Multi-source Unsupervised Domain Adaptation with Category Shift , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[15]  Taesung Park,et al.  CyCADA: Cycle-Consistent Adversarial Domain Adaptation , 2017, ICML.

[16]  Li Ma,et al.  Centroid and Covariance Alignment-Based Domain Adaptation for Unsupervised Classification of Remote Sensing Images , 2019, IEEE Transactions on Geoscience and Remote Sensing.

[17]  Lorenzo Bruzzone,et al.  Unsupervised retraining of a maximum likelihood classifier for the analysis of multitemporal remote sensing images , 2001, IEEE Trans. Geosci. Remote. Sens..

[18]  Chao Li,et al.  Active multi-kernel domain adaptation for hyperspectral image classification , 2017, Pattern Recognit..

[19]  Trevor Darrell,et al.  FCNs in the Wild: Pixel-level Adversarial and Constraint-based Adaptation , 2016, ArXiv.

[20]  Alex Bewley,et al.  Incremental Adversarial Domain Adaptation for Continually Changing Environments , 2017, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[21]  Lorenzo Bruzzone,et al.  Domain Adaptation for the Classification of Remote Sensing Data: An Overview of Recent Advances , 2016, IEEE Geoscience and Remote Sensing Magazine.

[22]  Pierre Alliez,et al.  ColorMapGAN: Unsupervised Domain Adaptation for Semantic Segmentation Using Color Mapping Generative Adversarial Networks , 2019, IEEE Transactions on Geoscience and Remote Sensing.

[23]  Pierre Alliez,et al.  SEMI2I: Semantically Consistent Image-to-Image Translation for Domain Adaptation of Remote Sensing Data , 2020, IGARSS 2020 - 2020 IEEE International Geoscience and Remote Sensing Symposium.

[24]  Maneesh Kumar Singh,et al.  DRIT++: Diverse Image-to-Image Translation via Disentangled Representations , 2019, International Journal of Computer Vision.

[25]  Jung-Woo Ha,et al.  StarGAN: Unified Generative Adversarial Networks for Multi-domain Image-to-Image Translation , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[26]  Luis Gómez-Chova,et al.  Graph Matching for Adaptation in Remote Sensing , 2013, IEEE Transactions on Geoscience and Remote Sensing.

[27]  Robert M. Haralick,et al.  Textural Features for Image Classification , 1973, IEEE Trans. Syst. Man Cybern..

[28]  Lorenzo Bruzzone,et al.  Domain Adaptation Problems: A DASVM Classification Technique and a Circular Validation Strategy , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[29]  Iasonas Kokkinos,et al.  DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[30]  Enrico Magli,et al.  Learning and Adapting Robust Features for Satellite Image Segmentation on Heterogeneous Data Sets , 2019, IEEE Transactions on Geoscience and Remote Sensing.

[31]  William J. Emery,et al.  The Importance of Physical Quantities for the Analysis of Multitemporal and Multiangular Optical Very High Spatial Resolution Images , 2014, IEEE Transactions on Geoscience and Remote Sensing.

[32]  拓海 杉山,et al.  “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”の学習報告 , 2017 .

[33]  Ming-Hsuan Yang,et al.  Learning to Adapt Structured Output Space for Semantic Segmentation , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[34]  Lorenzo Bruzzone,et al.  A multiple-cascade-classifier system for a robust and partially unsupervised updating of land-cover maps , 2002, IEEE Trans. Geosci. Remote. Sens..

[35]  Adam Van Etten,et al.  SpaceNet: A Remote Sensing Dataset and Challenge Series , 2018, ArXiv.

[36]  Bo Wang,et al.  Moment Matching for Multi-Source Domain Adaptation , 2018, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[37]  Shawki Areibi,et al.  Multisource Domain Adaptation for Remote Sensing Using Deep Neural Networks , 2020, IEEE Transactions on Geoscience and Remote Sensing.

[38]  M. Abidi,et al.  An Overview of Color Constancy Algorithms , 2006 .

[39]  Jan Kautz,et al.  Multimodal Unsupervised Image-to-Image Translation , 2018, ECCV.

[40]  Serge J. Belongie,et al.  Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[41]  Pierre Alliez,et al.  Convolutional Neural Networks for Large-Scale Remote-Sensing Image Classification , 2017, IEEE Transactions on Geoscience and Remote Sensing.

[42]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[43]  Vladimir Pavlovic,et al.  Unsupervised Multi-Target Domain Adaptation: An Information Theoretic Approach , 2018, IEEE Transactions on Image Processing.

[44]  Pierre Alliez,et al.  StandardGAN: Multi-source Domain Adaptation for Semantic Segmentation of Very High Resolution Satellite Images by Data Standardization , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[45]  Jian Yang,et al.  Online Adaptation through Meta-Learning for Stereo Depth Estimation , 2019, ArXiv.

[46]  Anis Koubaa,et al.  Unsupervised Domain Adaptation using Generative Adversarial Networks for Semantic Segmentation of Aerial Images , 2019, Remote. Sens..

[47]  Lorenzo Bruzzone,et al.  Active Learning for Domain Adaptation in the Supervised Classification of Remote Sensing Images , 2012, IEEE Transactions on Geoscience and Remote Sensing.

[48]  Jung-Woo Ha,et al.  StarGAN v2: Diverse Image Synthesis for Multiple Domains , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[49]  G. Buchsbaum A spatial processor model for object colour perception , 1980 .

[50]  José M. F. Moura,et al.  Adversarial Multiple Source Domain Adaptation , 2018, NeurIPS.

[51]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[52]  Yang Zou,et al.  Domain Adaptation for Semantic Segmentation via Class-Balanced Self-Training , 2018, ArXiv.

[53]  Lorenzo Bruzzone,et al.  A partially unsupervised cascade classifier for the analysis of multitemporal remote-sensing images , 2002, Pattern Recognit. Lett..

[54]  Huanhuan Yu,et al.  Multi-target Unsupervised Domain Adaptation without Exactly Shared Categories , 2018, ArXiv.

[55]  Jan Kautz,et al.  Unsupervised Image-to-Image Translation Networks , 2017, NIPS.

[56]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[57]  Francesca Bovolo,et al.  Multidimensional Probability Density Function Matching for Preprocessing of Multitemporal Remote Sensing Images , 2008, IEEE Transactions on Geoscience and Remote Sensing.

[58]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[59]  Klaus I. Itten,et al.  Geometric and radiometric correction of TM data of mountainous forested areas , 1993, IEEE Trans. Geosci. Remote. Sens..

[60]  David A. Forsyth,et al.  A novel algorithm for color constancy , 1990, International Journal of Computer Vision.

[61]  Nicolas Courty,et al.  Optimal Transport for Multi-source Domain Adaptation under Target Shift , 2018, AISTATS.