Semantic Segmentation of Remote Sensing Images Using Transfer Learning and Deep Convolutional Neural Network With Dense Connection

Semantic segmentation is an important approach in remote sensing image analysis. However, when segmenting multiobject from remote sensing images with insufficient labeled data and imbalanced data classes, the performances of the current semantic segmentation models were often unsatisfactory. In this paper, we try to solve this problem with transfer learning and a novel deep convolutional neural network with dense connection. We designed a UNet-based deep convolutional neural network, which is called TL-DenseUNet, for the semantic segmentation of remote sensing images. The proposed TL-DenseUNet contains two subnetworks. Among them, the encoder subnetwork uses a transferring DenseNet pretrained on three-band ImageNet images to extract multilevel semantic features, and the decoder subnetwork adopts dense connection to fuse the multiscale information in each layer, which can strengthen the expressive capability of the features. We carried out comprehensive experiments on remote sensing image datasets with 11 classes of ground objects. The experimental results demonstrate that both transfer learning and dense connection are effective for the multiobject semantic segmentation of remote sensing images with insufficient labeled data and imbalanced data classes. Compared with several other state-of-the-art models, the kappa coefficient of TL-DenseUNet is improved by more than 0.0752. TL-DenseUNet achieves better performance and more accurate segmentation results than the state-of-the-art models.

[1]  Navin Kumar Manaswi,et al.  Deep Learning with Applications Using Python , 2018, Apress.

[2]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Hongwei Liu,et al.  Convolutional Neural Network With Data Augmentation for SAR Target Recognition , 2016, IEEE Geoscience and Remote Sensing Letters.

[4]  Trevor Darrell,et al.  Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation , 2013, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[5]  Yongfeng Zhu,et al.  Semi-Supervised Deep Transfer Learning-Based on Adversarial Feature Learning for Label Limited SAR Target Recognition , 2019, IEEE Access.

[6]  Yoshua Bengio,et al.  How transferable are features in deep neural networks? , 2014, NIPS.

[7]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Xia Xu,et al.  R-VCANet: A New Deep-Learning-Based Hyperspectral Image Classification Method , 2017, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[9]  Ying Wang,et al.  Accurate urban road centerline extraction from VHR imagery via multiscale segmentation and tensor voting , 2015, Neurocomputing.

[10]  Peerapon Vateekul,et al.  Semantic Segmentation on Remotely Sensed Images Using an Enhanced Global Convolutional Network with Channel Attention and Domain Specific Transfer Learning , 2018, Remote. Sens..

[11]  Xiangyu Zhang,et al.  Large Kernel Matters — Improve Semantic Segmentation by Global Convolutional Network , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Roberto Cipolla,et al.  SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  Ian D. Reid,et al.  RefineNet: Multi-path Refinement Networks for High-Resolution Semantic Segmentation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Cristian R. Munteanu,et al.  Deep Learning Applications , 2018, Proceedings of MOL2NET 2018, International Conference on Multidisciplinary Sciences, 4th edition.

[15]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[16]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[18]  Kun Zhu,et al.  Symmetrical Dense-Shortcut Deep Fully Convolutional Networks for Semantic Segmentation of Very-High-Resolution Remote Sensing Images , 2018, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[19]  Leena Matikainen,et al.  Segment-Based Land Cover Mapping of a Suburban Area - Comparison of High-Resolution Remotely Sensed Datasets Using Classification Trees and Test Field Points , 2011, Remote. Sens..

[20]  Ning Zhang,et al.  CoinNet: Copy Initialization Network for Multispectral Imagery Semantic Segmentation , 2019, IEEE Geoscience and Remote Sensing Letters.

[21]  Ivan Laptev,et al.  Learning and Transferring Mid-level Image Representations Using Convolutional Neural Networks , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[22]  Jamie Sherrah,et al.  Fully Convolutional Networks for Dense Semantic Labelling of High-Resolution Aerial Imagery , 2016, ArXiv.

[23]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[24]  Yuan Yu,et al.  TensorFlow: A system for large-scale machine learning , 2016, OSDI.

[25]  Stefan Carlsson,et al.  CNN Features Off-the-Shelf: An Astounding Baseline for Recognition , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops.

[26]  Shiyong Cui,et al.  Building Footprint Extraction From VHR Remote Sensing Images Combined With Normalized DSMs Using Fused Fully Convolutional Networks , 2018, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[27]  Iasonas Kokkinos,et al.  DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[28]  Rogério Schmidt Feris,et al.  SpotTune: Transfer Learning Through Adaptive Fine-Tuning , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  Toby P. Breckon,et al.  Transfer learning using convolutional neural networks for object classification within X-ray baggage security imagery , 2016, 2016 IEEE International Conference on Image Processing (ICIP).

[30]  Yongyang Xu,et al.  Building Extraction in Very High Resolution Remote Sensing Imagery Using Deep Learning and Guided Filters , 2018, Remote. Sens..

[31]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[32]  Zhenwei Shi,et al.  Automatic Raft Labeling for Remote Sensing Images via Dual-Scale Homogeneous Convolutional Neural Network , 2018, Remote. Sens..

[33]  Lingfeng Wang,et al.  Semantic Labeling in Very High Resolution Images via a Self-Cascaded Convolutional Neural Network , 2017, ISPRS Journal of Photogrammetry and Remote Sensing.

[34]  Xia Li,et al.  Road Detection From Remote Sensing Images by Generative Adversarial Networks , 2018, IEEE Access.

[35]  Yoshua Bengio,et al.  Deep Sparse Rectifier Neural Networks , 2011, AISTATS.

[36]  Meng Lan,et al.  Global context based automatic road segmentation via dilated convolutional neural network , 2020, Inf. Sci..

[37]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[38]  Graham W. Taylor,et al.  Adaptive deconvolutional networks for mid and high level feature learning , 2011, 2011 International Conference on Computer Vision.

[39]  Michael I. Jordan,et al.  Learning Transferable Features with Deep Adaptation Networks , 2015, ICML.

[40]  Xiao Xiang Zhu,et al.  Deep Learning in Remote Sensing: A Comprehensive Review and List of Resources , 2017, IEEE Geoscience and Remote Sensing Magazine.

[41]  Xueliang Zhang,et al.  Deep learning in remote sensing applications: A meta-analysis and review , 2019, ISPRS Journal of Photogrammetry and Remote Sensing.

[42]  Xiaocong Xu,et al.  Building Footprint Extraction from High-Resolution Images via Spatial Residual Inception Convolutional Neural Network , 2019, Remote. Sens..

[43]  Sethuraman Panchanathan,et al.  Active Batch Selection via Convex Relaxations with Guaranteed Solution Bounds , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[44]  Tian Zhao,et al.  Semantic Segmentation of Urban Buildings from VHR Remote Sensing Imagery Using a Deep Convolutional Neural Network , 2019, Remote. Sens..

[45]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[46]  Bo Du,et al.  Deep Learning for Remote Sensing Data: A Technical Tutorial on the State of the Art , 2016, IEEE Geoscience and Remote Sensing Magazine.

[47]  José Luis Hernández-Stefanoni,et al.  Predicting old-growth tropical forest attributes from very high resolution (VHR)-derived surface metrics , 2017 .

[48]  Uwe Stilla,et al.  Deep Learning Earth Observation Classification Using ImageNet Pretrained Networks , 2016, IEEE Geoscience and Remote Sensing Letters.