A Building Segmentation Network Based on Improved Spatial Pyramid in Remote Sensing Images

Building segmentation is widely used in urban planning, disaster prevention, human flow monitoring and environmental monitoring. However, due to the complex landscapes and highdensity settlements, automatically characterizing building in the urban village or cities using remote sensing images is very challenging. Inspired by the rencent deep learning methods, this paper proposed a novel end-to-end building segmentation network for segmenting buildings from remote sensing images. The network includes two branches: one branch uses Widely Adaptive Spatial Pyramid (WASP) structure to extract multi-scale features, and the other branch uses a deep residual network combined with a sub-pixel up-sampling structure to enhance the detail of building boundaries. We compared our proposed method with three state-of-the-art networks: DeepLabv3+, ENet, ESPNet. Experiments were performed using the publicly available Inria Aerial Image Labelling dataset (Inria aerial dataset) and the Satellite dataset II(East Asia). The results showed that our method outperformed the other networks in the experiments, with Pixel Accuracy reaching 0.8421 and 0.8738, respectively and with mIoU reaching 0.9034 and 0.8936 respectively. Compared with the basic network, it has increased by about 25% or more. It can not only extract building footprints, but also especially small building objects.

[1]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[2]  Wei Liu,et al.  Ship Classification and Detection Based on CNN Using GF-3 SAR Images , 2018, Remote. Sens..

[3]  D. Zaum,et al.  ROBUST BUILDING DETECTION IN AERIAL IMAGES , 2005 .

[4]  Antonio J. Plaza,et al.  Semisupervised Hyperspectral Image Segmentation Using Multinomial Logistic Regression With Active Learning , 2010, IEEE Transactions on Geoscience and Remote Sensing.

[5]  Zhengjun Liu,et al.  Building extraction from high resolution satellite imagery based on multi-scale image segmentation and model matching , 2008, 2008 International Workshop on Earth Observation and Remote Sensing Applications.

[6]  Garrison W. Cottrell,et al.  Understanding Convolution for Semantic Segmentation , 2017, 2018 IEEE Winter Conference on Applications of Computer Vision (WACV).

[7]  Wei Wang,et al.  Deep Learning for Single Image Super-Resolution: A Brief Review , 2018, IEEE Transactions on Multimedia.

[8]  Daniel Rueckert,et al.  Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Mohammad Awrangjeb,et al.  Using point cloud data to identify, trace, and regularize the outlines of buildings , 2016 .

[10]  G LoweDavid,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004 .

[11]  Enhua Wu,et al.  Squeeze-and-Excitation Networks , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  Stanley Osher,et al.  Image Super-Resolution by TV-Regularization and Bregman Iteration , 2008, J. Sci. Comput..

[13]  Lorenzo Bruzzone,et al.  Classification of hyperspectral remote sensing images with support vector machines , 2004, IEEE Transactions on Geoscience and Remote Sensing.

[14]  Chris H. Q. Ding,et al.  Semi-Supervised Remote Sensing Image Semantic Segmentation via Consistency Regularization and Average Update of Pseudo-Label , 2020, Remote. Sens..

[15]  Eugenio Culurciello,et al.  ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation , 2016, ArXiv.

[16]  Xiang Bai,et al.  EPNet: Enhancing Point Features with Image Semantics for 3D Object Detection , 2020, ECCV.

[17]  Hamid Abrishami Moghaddam,et al.  Automatic urban building boundary extraction from high resolution aerial images using an innovative model of active contours , 2010, Int. J. Appl. Earth Obs. Geoinformation.

[18]  Qian Zhang,et al.  Urban Area Extraction by Regional and Line Segment Feature Fusion and Urban Morphology Analysis , 2017, Remote. Sens..

[19]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[20]  Shiyong Cui,et al.  BUILDING EXTRACTION FROM REMOTE SENSING DATA USING FULLY CONVOLUTIONAL NETWORKS , 2017 .

[21]  Paul M. Mather,et al.  An assessment of the effectiveness of decision tree methods for land cover classification , 2003 .

[22]  Yinghai Ke,et al.  Urban Land Use and Land Cover Classification Using Novel Deep Learning Models Based on High Spatial Resolution Satellite Imagery , 2018, Sensors.

[23]  Yi Zhang,et al.  WSF-NET: Weakly Supervised Feature-Fusion Network for Binary Segmentation in Remote Sensing Image , 2018, Remote. Sens..

[24]  Kyoung Mu Lee,et al.  Accurate Image Super-Resolution Using Very Deep Convolutional Networks , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Bo Li,et al.  Automatic Ship Detection in Optical Remote Sensing Images Based on Anomaly Detection and SPP-PCANet , 2018, Remote. Sens..

[26]  Jing Peng,et al.  An improved snake model for building detection from urban aerial images , 2005, Pattern Recognit. Lett..

[27]  Xiaoou Tang,et al.  Image Super-Resolution Using Deep Convolutional Networks , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[28]  Ming Hao,et al.  Building Change Detection Using a Shape Context Similarity Model for LiDAR Data , 2020, ISPRS Int. J. Geo Inf..

[29]  Mei Han,et al.  SoftCuts: A Soft Edge Smoothness Prior for Color Image Super-Resolution , 2009, IEEE Transactions on Image Processing.

[30]  Xin Huang,et al.  An automatic change detection method for monitoring newly constructed building areas using time-series multi-view high-resolution optical satellite images , 2020 .

[31]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  Leonardo Vanneschi,et al.  Improved Fully Convolutional Network with Conditional Random Fields for Building Extraction , 2018, Remote. Sens..

[33]  Li Fei-Fei,et al.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution , 2016, ECCV.

[34]  Jie Chen,et al.  SPMF-Net: Weakly Supervised Building Segmentation by Combining Superpixel Pooling and Multi-Scale Feature Fusion , 2020, Remote. Sens..

[35]  Curt H. Davis,et al.  Automated 2-D building footprint extraction from high-resolution satellite multispectral imagery , 2004, IGARSS 2004. 2004 IEEE International Geoscience and Remote Sensing Symposium.

[36]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[37]  Truong Q. Nguyen,et al.  Single Image Superresolution Based on Gradient Profile Sharpness , 2015, IEEE Transactions on Image Processing.

[38]  Giles M. Foody,et al.  An evaluation of some factors affecting the accuracy of classification by an artificial neural network , 1997 .

[39]  Yuhao Wang,et al.  Dense Semantic Labeling with Atrous Spatial Pyramid Pooling and Decoder for High-Resolution Remote Sensing Imagery , 2018, Remote. Sens..

[40]  Yun Fu,et al.  Residual Dense Network for Image Super-Resolution , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[41]  George Papandreou,et al.  Rethinking Atrous Convolution for Semantic Image Segmentation , 2017, ArXiv.

[42]  Wei Yuan,et al.  Automatic Building Segmentation of Aerial Imagery Using Multi-Constraint Fully Convolutional Networks , 2018, Remote. Sens..

[43]  Antonio J. Plaza,et al.  This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination. IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING 1 Spectral–Spatial Hyperspectral Image Segmentation Using S , 2022 .

[44]  Pengfei Xiong,et al.  Pyramid Attention Network for Semantic Segmentation , 2018, BMVC.

[45]  Kyoung Mu Lee,et al.  Deeply-Recursive Convolutional Network for Image Super-Resolution , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[46]  Xiaogang Wang,et al.  Pyramid Scene Parsing Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[47]  Lan Chen,et al.  Knowle: A semantic link network based system for organizing large scale online news events , 2015, Future Gener. Comput. Syst..

[48]  Vladlen Koltun,et al.  Multi-Scale Context Aggregation by Dilated Convolutions , 2015, ICLR.

[49]  Yongyang Xu,et al.  Building Extraction in Very High Resolution Remote Sensing Imagery Using Deep Learning and Guided Filters , 2018, Remote. Sens..

[50]  Marcela Gergeľová,et al.  Identification of Roof Surfaces from LiDAR Cloud Points by GIS Tools: A Case Study of Lučenec, Slovakia , 2020, Sustainability.

[51]  Luc Van Gool,et al.  Speeded-Up Robust Features (SURF) , 2008, Comput. Vis. Image Underst..

[52]  Xin Huang,et al.  Deep neural network for remote-sensing image interpretation: status and perspectives , 2019, National science review.

[53]  Kyoung Mu Lee,et al.  Enhanced Deep Residual Networks for Single Image Super-Resolution , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[54]  Liangpei Zhang,et al.  An Adaptive Artificial Immune Network for Supervised Classification of Multi-/Hyperspectral Remote Sensing Imagery , 2012, IEEE Transactions on Geoscience and Remote Sensing.

[55]  P. Atkinson,et al.  Introduction Neural networks in remote sensing , 1997 .

[56]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[57]  Razvan Pascanu,et al.  On the Number of Linear Regions of Deep Neural Networks , 2014, NIPS.

[58]  Ying Wang,et al.  Gated Convolutional Neural Network for Semantic Segmentation in High-Resolution Images , 2017, Remote. Sens..

[59]  Xiaoou Tang,et al.  Learning a Deep Convolutional Network for Image Super-Resolution , 2014, ECCV.

[60]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[61]  Chun Liu,et al.  Automatic extraction of built-up area from ZY3 multi-view satellite imagery: Analysis of 45 global cities , 2019, Remote Sensing of Environment.

[62]  Iasonas Kokkinos,et al.  DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[63]  Iasonas Kokkinos,et al.  Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs , 2014, ICLR.

[64]  Xiaogang Wang,et al.  Context Encoding for Semantic Segmentation , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.