Cross-domain road detection based on global-local adversarial learning framework from very high resolution satellite imagery

Abstract Road detection based on convolutional neural networks (CNNs) has achieved remarkable performances for very high resolution (VHR) remote sensing images. However, this approach relies on massive annotated samples, and the problem of limited generalization for unseen images still remains. The manual pixel-level labeling process is also extremely time-consuming, and the performance of CNNs degrades significantly when there is a domain gap between the training and test images. In this paper, to address this problem, a global-local adversarial learning (GOAL) framework is proposed for cross-domain road detection. On the one hand, considering the spatial information similarities between the source and target domains, feature space driven adversarial learning is applied to explore the shared features across domains. On the other hand, the complex background of VHR remote sensing images, such as the occlusions and shadows of trees and buildings, makes some roads easy to recognize, while others are much more difficult. However, the traditional global adversarial learning approach cannot guarantee local semantic consistency. Therefore, a local alignment operation is introduced, which adaptively adjusts the weight of the adversarial loss according to the road recognition difficulty. Extensive experiments were conducted on different road datasets, including two public competition road datasets—SpaceNet and DeepGlobe—and our own large-scale annotated images from four cities: Boston, Birmingham, Shanghai, and Wuhan. The experimental results show that the proposed GOAL framework can clearly improve the cross-domain road detection performance, without any annotation of the target domain images. For instance, taking SpaceNet road dataset as the source domain, compared with the no adaptation method, the IOU performance of GOAL framework is increased by 14.36%, 5.49%,  4.51%, 5.63% and 15.14% on DeepGlobe, Boston, Birmingham, Shanghai, and Wuhan images, respectively, which demonstrates its strong generalization capability.

[1]  Léon Bottou,et al.  Large-Scale Machine Learning with Stochastic Gradient Descent , 2010, COMPSTAT.

[2]  Chun Liu,et al.  Leveraging Crowdsourced GPS Data for Road Extraction From Aerial Imagery , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[4]  Geoffrey E. Hinton,et al.  Machine Learning for Aerial Image Labeling , 2013 .

[5]  拓海 杉山,et al.  “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”の学習報告 , 2017 .

[6]  Ji Zhao,et al.  Edge-Reinforced Convolutional Neural Network for Road Detection in Very-High-Resolution Remote Sensing Imagery , 2020 .

[7]  Richard Lepage,et al.  Road Extraction From Very High Resolution Remote Sensing Optical Images Based on Texture Analysis and Beamlet Transform , 2016, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[8]  Nuno Vasconcelos,et al.  Bidirectional Learning for Domain Adaptation of Semantic Segmentation , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Ming-Hsuan Yang,et al.  Learning to Adapt Structured Output Space for Semantic Segmentation , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[10]  Jinjun Xiong,et al.  Differential Treatment for Stuff and Things: A Simple Unsupervised Domain Adaptation Method for Semantic Segmentation , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Jian Yao,et al.  RoadNet: Learning to Comprehensively Analyze Road Networks in Complex Urban Scenes From High-Resolution Remotely Sensed Images , 2019, IEEE Transactions on Geoscience and Remote Sensing.

[12]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[13]  Wenzhong Shi,et al.  The line segment match method for extracting road network from high-resolution satellite images , 2002, IEEE Trans. Geosci. Remote. Sens..

[14]  Ji Zhao,et al.  HyNet: Hyper-scale object detection network framework for multiple spatial resolution remote sensing imagery , 2020 .

[15]  Iasonas Kokkinos,et al.  DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[16]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Zhuo Zheng,et al.  GAMSNet: Globally aware road detection network with multi-scale residual learning , 2021 .

[18]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[19]  Jing Huang,et al.  DeepGlobe 2018: A Challenge to Parse the Earth through Satellite Images , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[20]  Luc Van Gool,et al.  Learning Semantic Segmentation From Synthetic Data: A Geometrically Guided Input-Output Adaptation Approach , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Ivan Laptev,et al.  Learning and Transferring Mid-level Image Representations Using Convolutional Neural Networks , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[22]  Hui Xiong,et al.  A Unified Framework for Metric Transfer Learning , 2017, IEEE Transactions on Knowledge and Data Engineering.

[23]  Ignacio Parra,et al.  Deep fully convolutional networks with random data augmentation for enhanced generalization in road detection , 2017, 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC).

[24]  Haifeng Li,et al.  Spatial Information Inference Net: Road Extraction Using Road-Specific Contextual Information , 2019, IGARSS 2019 - 2019 IEEE International Geoscience and Remote Sensing Symposium.

[25]  Aigong Xu,et al.  Automatic Extraction of Urban Road Centerlines from High-Resolution Satellite Imagery Using Automatic Thresholding and Morphological Operation Method , 2016 .

[26]  Yue Zhao,et al.  Integrated use of spatial and semantic relationships for extracting road networks from floating car data , 2012, Int. J. Appl. Earth Obs. Geoinformation.

[27]  Liangpei Zhang,et al.  An SVM Ensemble Approach Combining Spectral, Structural, and Semantic Features for the Classification of High-Resolution Remotely Sensed Imagery , 2013, IEEE Transactions on Geoscience and Remote Sensing.

[28]  Wei-Lun Chang,et al.  All About Structure: Adapting Structural Information Across Domains for Boosting Semantic Segmentation , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[30]  Chao Yang,et al.  A Survey on Deep Transfer Learning , 2018, ICANN.

[31]  Qingjie Liu,et al.  Road Extraction by Deep Residual U-Net , 2017, IEEE Geoscience and Remote Sensing Letters.

[32]  Zhi-Hua Zhou,et al.  Tri-training: exploiting unlabeled data using three classifiers , 2005, IEEE Transactions on Knowledge and Data Engineering.

[33]  Yuning Jiang,et al.  DeepDualMapper: A Gated Fusion Network for Automatic Map Extraction using Aerial Images and Trajectories , 2020, AAAI.

[34]  Pietro Zanuttigh,et al.  Unsupervised Domain Adaptation in Semantic Segmentation: a Review , 2020, ArXiv.

[35]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[36]  Andrew L. Maas.  Rectifier Nonlinearities Improve Neural Network Acoustic Models , 2013 .

[37]  Wenzhong Shi,et al.  A comprehensive quality assessment framework for linear features from Volunteered Geographic Information , 2020, Int. J. Geogr. Inf. Sci..

[38]  Raquel Urtasun,et al.  DeepRoadMapper: Extracting Road Topology from Aerial Images , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[39]  Vincent Frémont,et al.  Exploiting fully convolutional neural networks for fast road detection , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[40]  Patrick Pérez,et al.  ADVENT: Adversarial Entropy Minimization for Domain Adaptation in Semantic Segmentation , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[41]  Nikhil Ketkar,et al.  Introduction to PyTorch , 2021, Deep Learning with Python.

[42]  Zhengjun Liu,et al.  Semi-automatic road tracking by template matching and distance transformation in urban areas , 2011 .

[43]  Liang Chen,et al.  A Fully Convolutional Tri-Branch Network (FCTN) for Domain Adaptation , 2018, 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[44]  Jie Yang,et al.  Multi-Scale and Multi-Task Deep Learning Framework for Automatic Road Extraction , 2019, IEEE Transactions on Geoscience and Remote Sensing.

[45]  H. Alemohammad,et al.  Detecting Roads from Satellite Imagery in the Developing World , 2019, CVPR Workshops.

[46]  Ming Wu,et al.  D-LinkNet: LinkNet with Pretrained Encoder and Dilated Convolution for High Resolution Satellite Imagery Road Extraction , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[47]  Lorenzo Bruzzone,et al.  Semantic Segmentation of Large-Size VHR Remote Sensing Images Using a Two-Stage Multiscale Training Architecture , 2020, IEEE Transactions on Geoscience and Remote Sensing.

[48]  Ryosuke Shibasaki,et al.  Semi-automatic road extraction from high-resolution satellite image , 2002 .

[49]  Peter Wonka,et al.  Road Network Extraction and Intersection Detection From Aerial Images by Tracking Road Footprints , 2007, IEEE Transactions on Geoscience and Remote Sensing.

[50]  C. V. Jawahar,et al.  Improved Road Connectivity by Joint Learning of Orientation and Segmentation , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[51]  Jian-Jiun Ding,et al.  Accurate Road Detection from Satellite Images Using Modified U-net , 2018, 2018 IEEE Asia Pacific Conference on Circuits and Systems (APCCAS).

[52]  David J. DeWitt,et al.  RoadTracer: Automatic Extraction of Road Networks from Aerial Images , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[53]  Zhuo Zheng,et al.  Foreground-Aware Relation Network for Geospatial Object Segmentation in High Spatial Resolution Remote Sensing Imagery , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[54]  Anis Koubaa,et al.  Unsupervised Domain Adaptation using Generative Adversarial Networks for Semantic Segmentation of Aerial Images , 2019, Remote. Sens..