Land-Cover Classification with High-Resolution Remote Sensing Images Using Transferable Deep Models

Abstract In recent years, large amount of high spatial-resolution remote sensing (HRRS) images are available for land-cover mapping. However, due to the complex information brought by the increased spatial resolution and the data disturbances caused by different conditions of image acquisition, it is often difficult to find an efficient method for achieving accurate land-cover classification with high-resolution and heterogeneous remote sensing images. In this paper, we propose a scheme to apply deep model obtained from labeled land-cover dataset to classify unlabeled HRRS images. The main idea is to rely on deep neural networks for presenting the contextual information contained in different types of land-covers and propose a pseudo-labeling and sample selection scheme for improving the transferability of deep models. More precisely, a deep Convolutional Neural Networks (CNNs) is first pre-trained with a well-annotated land-cover dataset, referred to as the source data. Then, given a target image with no labels, the pre-trained CNN model is utilized to classify the image in a patch-wise manner. The patches with high confidence are assigned with pseudo-labels and employed as the queries to retrieve related samples from the source data. The pseudo-labels confirmed with the retrieved results are regarded as supervised information for fine-tuning the pre-trained deep model. To obtain a pixel-wise land-cover classification with the target image, we rely on the fine-tuned CNN and develop a hybrid classification by combining patch-wise classification and hierarchical segmentation. In addition, we create a large-scale land-cover dataset containing 150 Gaofen-2 satellite images for CNN pre-training. Experiments on multi-source HRRS images, including Gaofen-2, Gaofen-1, Jilin-1, Ziyuan-3, Sentinel-2A, and Google Earth platform data, show encouraging results and demonstrate the applicability of the proposed scheme to land-cover classification with multi-source HRRS images.

[1]  Jon Atli Benediktsson,et al.  Segmentation and Classification of Hyperspectral Images Using Minimum Spanning Forest Grown From Automatically Selected Markers , 2010, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[2]  Zhou Guo,et al.  On combining multiscale deep learning features for the classification of hyperspectral remote sensing imagery , 2015 .

[3]  Dong-Hyun Lee,et al.  Pseudo-Label : The Simple and Efficient Semi-Supervised Learning Method for Deep Neural Networks , 2013 .

[4]  Uwe Stilla,et al.  Deep Learning Earth Observation Classification Using ImageNet Pretrained Networks , 2016, IEEE Geoscience and Remote Sensing Letters.

[5]  Nataliia Kussul,et al.  Regional scale crop mapping using multi-temporal satellite imagery , 2015 .

[6]  P. Soille,et al.  Information extraction from very high resolution satellite imagery over Lukole refugee camp, Tanzania , 2003 .

[7]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[8]  Bertrand Le Saux,et al.  How useful is region-based classification of remote sensing images in a deep learning framework? , 2016, 2016 IEEE International Geoscience and Remote Sensing Symposium (IGARSS).

[9]  Xin Pan,et al.  An object-based convolutional neural network (OCNN) for urban land use classification , 2018, Remote Sensing of Environment.

[10]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[11]  Gui-Song Xia,et al.  Learning High-level Features for Satellite Image Classification With Limited Labeled Samples , 2015, IEEE Transactions on Geoscience and Remote Sensing.

[12]  Gustavo Camps-Valls,et al.  Multisource Composite Kernels for Urban-Image Classification , 2010, IEEE Geoscience and Remote Sensing Letters.

[13]  Sethuraman Panchanathan,et al.  Active Batch Selection via Convex Relaxations with Guaranteed Solution Bounds , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  Gui-Song Xia,et al.  AID: A Benchmark Data Set for Performance Evaluation of Aerial Scene Classification , 2016, IEEE Transactions on Geoscience and Remote Sensing.

[15]  Kim-Hui Yap,et al.  Fuzzy SVM for content-based image retrieval: a pseudo-label support vector machine framework , 2006, IEEE Computational Intelligence Magazine.

[16]  Gang Liu,et al.  Texture Characterization Using Shape Co-Occurrence Patterns. , 2017, IEEE transactions on image processing : a publication of the IEEE Signal Processing Society.

[17]  Daniel P. Huttenlocher,et al.  Efficient Graph-Based Image Segmentation , 2004, International Journal of Computer Vision.

[18]  Goo Jun,et al.  Spatially Adaptive Classification of Land Cover With Remote Sensing Data , 2011, IEEE Transactions on Geoscience and Remote Sensing.

[19]  Pilar Casals-Carrasco,et al.  Application of spectral mixture analysis for terrain evaluation studies , 2000 .

[20]  Jon Atli Benediktsson,et al.  Land-Cover Mapping by Markov Modeling of Spatial–Contextual Information in Very-High-Resolution Remote Sensing Images , 2013, Proceedings of the IEEE.

[21]  Gui-Song Xia,et al.  Deep sparse representations for land-use scene classification in remote sensing images , 2016, 2016 IEEE 13th International Conference on Signal Processing (ICSP).

[22]  Xin Pan,et al.  A hybrid MLP-CNN classifier for very fine resolution remotely sensed image classification , 2017, ISPRS Journal of Photogrammetry and Remote Sensing.

[23]  Jon Atli Benediktsson,et al.  SVM- and MRF-Based Method for Accurate Classification of Hyperspectral Images , 2010, IEEE Geoscience and Remote Sensing Letters.

[24]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Lorenzo Bruzzone,et al.  Domain Adaptation for the Classification of Remote Sensing Data: An Overview of Recent Advances , 2016, IEEE Geoscience and Remote Sensing Magazine.

[26]  Gui-Song Xia,et al.  Dirichlet-Derived Multiple Topic Scene Classification Model for High Spatial Resolution Remote Sensing Imagery , 2016, IEEE Transactions on Geoscience and Remote Sensing.

[27]  Peijun Du,et al.  A review of supervised object-based land-cover image classification , 2017 .

[28]  Gui-Song Xia,et al.  Fast Binary Coding for the Scene Classification of High-Resolution Remote Sensing Imagery , 2016, Remote. Sens..

[29]  Lorenzo Bruzzone,et al.  Active and Semisupervised Learning for the Classification of Remote Sensing Images , 2014, IEEE Transactions on Geoscience and Remote Sensing.

[30]  Francesca Bovolo,et al.  Detection of Land-Cover Transitions in Multitemporal Remote Sensing Images With Active-Learning-Based Compound Classification , 2012, IEEE Transactions on Geoscience and Remote Sensing.

[31]  Gui-Song Xia,et al.  Extreme value theory-based calibration for the fusion of multiple features in high-resolution satellite scene classification , 2013 .

[32]  Shihong Du,et al.  Learning multiscale and deep representations for classifying remotely sensed imagery , 2016 .

[33]  Jiebo Luo,et al.  DOTA: A Large-Scale Dataset for Object Detection in Aerial Images , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[34]  Peng Gong,et al.  A comparison of spatial feature extraction algorithms for land-use classification with SPOT HRV data , 1992 .

[35]  Xiao Xiang Zhu,et al.  Deep Learning in Remote Sensing: A Comprehensive Review and List of Resources , 2017, IEEE Geoscience and Remote Sensing Magazine.

[36]  Yu Liu,et al.  Hourglass-ShapeNetwork Based Semantic Segmentation for High Resolution Aerial Imagery , 2017, Remote. Sens..

[37]  Michele Volpi,et al.  Dense Semantic Labeling of Subdecimeter Resolution Images With Convolutional Neural Networks , 2016, IEEE Transactions on Geoscience and Remote Sensing.

[38]  Gui-Song Xia,et al.  Large-Scale Land Cover Classification in Gaofen-2 Satellite Imagery , 2018, IGARSS 2018 - 2018 IEEE International Geoscience and Remote Sensing Symposium.

[39]  Matti Pietikäinen,et al.  Multiresolution Gray-Scale and Rotation Invariant Texture Classification with Local Binary Patterns , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[40]  Robert M. Haralick,et al.  Textural Features for Image Classification , 1973, IEEE Trans. Syst. Man Cybern..

[41]  Iasonas Kokkinos,et al.  DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[42]  Naif Alajlan,et al.  Domain Adaptation Network for Cross-Scene Classification , 2017, IEEE Transactions on Geoscience and Remote Sensing.

[43]  Lorenzo Bruzzone,et al.  Active Learning for Domain Adaptation in the Supervised Classification of Remote Sensing Images , 2012, IEEE Transactions on Geoscience and Remote Sensing.

[44]  Jon Atli Benediktsson,et al.  Advances in Spectral-Spatial Classification of Hyperspectral Images , 2013, Proceedings of the IEEE.

[45]  Valentyn Tolpekin,et al.  Markov random field based super-resolution mapping for identification of urban trees in VHR images , 2010, 2010 IEEE International Geoscience and Remote Sensing Symposium.

[46]  Lorenzo Bruzzone,et al.  Semisupervised Transfer Component Analysis for Domain Adaptation in Remote Sensing Image Classification , 2015, IEEE Transactions on Geoscience and Remote Sensing.

[47]  Lorenzo Bruzzone,et al.  A Novel Transductive SVM for Semisupervised Classification of Remote-Sensing Images , 2006, IEEE Transactions on Geoscience and Remote Sensing.

[48]  Raj Acharya,et al.  Color clustering techniques for color-content-based image retrieval from image databases , 1997, Proceedings of IEEE International Conference on Multimedia Computing and Systems.

[49]  Nataliia Kussul,et al.  Deep Learning Classification of Land Cover and Crop Types Using Remote Sensing Data , 2017, IEEE Geoscience and Remote Sensing Letters.

[50]  Ying Yu,et al.  Accurate Urban Area Detection in Remote Sensing Images , 2015, IEEE Geoscience and Remote Sensing Letters.

[51]  Weiming Shen,et al.  Retrieving Aerial Scene Images with Learned Deep Image-Sketch Features , 2017, Journal of Computer Science and Technology.

[52]  Emma Izquierdo-Verdiguier,et al.  Encoding Invariances in Remote Sensing Image Classification With SVM , 2013, IEEE Geoscience and Remote Sensing Letters.

[53]  Patricia Gober,et al.  Per-pixel vs. object-based classification of urban land cover extraction using high spatial resolution imagery , 2011, Remote Sensing of Environment.

[54]  Giles M. Foody,et al.  Good practices for estimating area and assessing accuracy of land change , 2014 .

[55]  Jamie Sherrah,et al.  Fully Convolutional Networks for Dense Semantic Labelling of High-Resolution Aerial Imagery , 2016, ArXiv.

[56]  Rob Fergus,et al.  Visualizing and Understanding Convolutional Networks , 2013, ECCV.

[57]  Jun Li,et al.  A Novel MRF-Based Multifeature Fusion for Classification of Remote Sensing Images , 2016, IEEE Geoscience and Remote Sensing Letters.

[58]  Fei-Fei Li,et al.  ImageNet: A large-scale hierarchical image database , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[59]  Geoffrey E. Hinton,et al.  Machine Learning for Aerial Image Labeling , 2013 .

[60]  Jamie Sherrah,et al.  Semantic Labeling of Aerial and Satellite Imagery , 2016, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[61]  Andrew Zisserman,et al.  Return of the Devil in the Details: Delving Deep into Convolutional Nets , 2014, BMVC.

[62]  Roberto Cipolla,et al.  SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[63]  Chunhua Zhang,et al.  The application of small unmanned aerial systems for precision agriculture: a review , 2012, Precision Agriculture.

[64]  John R. Jensen,et al.  Introductory Digital Image Processing: A Remote Sensing Perspective , 1986 .

[65]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[66]  Gang Liu,et al.  A Color-Texture-Structure Descriptor for High-Resolution Satellite Image Classification , 2016, Remote. Sens..

[67]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[68]  Konrad Schindler,et al.  Mapping of Agricultural Crops from Single High-Resolution Multispectral Images - Data-Driven Smoothing vs. Parcel-Based Smoothing , 2015, Remote. Sens..

[69]  Xue Li,et al.  Iterative Reweighting Heterogeneous Transfer Learning Framework for Supervised Remote Sensing Image Classification , 2017, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[70]  Bo Huang,et al.  Transfer Learning With Fully Pretrained Deep Convolution Networks for Land-Use Classification , 2017, IEEE Geoscience and Remote Sensing Letters.

[71]  Mihai Datcu,et al.  Exploiting Deep Features for Remote Sensing Image Retrieval: A Systematic Investigation , 2017, IEEE Transactions on Big Data.

[72]  Koen E. A. van de Sande,et al.  Selective Search for Object Recognition , 2013, International Journal of Computer Vision.

[73]  Feng Gao,et al.  Representative lake water extent mapping at continental scales using multi-temporal Landsat-8 imagery , 2016 .

[74]  Paolo Napoletano,et al.  Visual descriptors for content-based retrieval of remote-sensing images , 2016, ArXiv.

[75]  Alfred Stein,et al.  Deep Fully Convolutional Networks for the Detection of Informal Settlements in VHR Images , 2017, IEEE Geoscience and Remote Sensing Letters.

[76]  Liangpei Zhang,et al.  A Hybrid Object-Oriented Conditional Random Field Classification Framework for High Spatial Resolution Remote Sensing Imagery , 2014, IEEE Transactions on Geoscience and Remote Sensing.

[77]  Zhang Xiangmin,et al.  Comparison of pixel‐based and object‐oriented image classification approaches—a case study in a coal fire area, Wuda, Inner Mongolia, China , 2006 .

[78]  Luis Gómez-Chova,et al.  Semisupervised Image Classification With Laplacian Support Vector Machines , 2008, IEEE Geoscience and Remote Sensing Letters.

[79]  Lorenzo Bruzzone,et al.  A Novel Approach to the Selection of Spatially Invariant Features for the Classification of Hyperspectral Images With Improved Generalization Capability , 2009, IEEE Transactions on Geoscience and Remote Sensing.

[80]  Wen Yang,et al.  STRUCTURAL HIGH-RESOLUTION SATELLITE IMAGE INDEXING , 2010 .

[81]  Lorenzo Bruzzone,et al.  Definition of Effective Training Sets for Supervised Classification of Remote Sensing Images by a Novel Cost-Sensitive Active Learning Method , 2014, IEEE Transactions on Geoscience and Remote Sensing.

[82]  Gustau Camps-Valls,et al.  Kernel Manifold Alignment for Domain Adaptation , 2015, PloS one.

[83]  Johannes R. Sveinsson,et al.  Classification of hyperspectral data from urban areas based on extended morphological profiles , 2005, IEEE Transactions on Geoscience and Remote Sensing.

[84]  Pierre Alliez,et al.  Convolutional Neural Networks for Large-Scale Remote-Sensing Image Classification , 2017, IEEE Transactions on Geoscience and Remote Sensing.

[85]  Bo Huang,et al.  Urban land-use mapping using a deep convolutional neural network with high spatial resolution multispectral remote sensing imagery , 2018, Remote Sensing of Environment.

[86]  Clement Atzberger,et al.  How much does multi-temporal Sentinel-2 data improve crop type classification? , 2018, Int. J. Appl. Earth Obs. Geoinformation.

[87]  Bei Zhao,et al.  Scene Semantic Understanding Based on the Spatial Context Relations of Multiple Objects , 2017, Remote. Sens..

[88]  Steven E. Franklin,et al.  A comparison of pixel-based and object-based image analysis with selected machine learning algorithms for the classification of agricultural landscapes using SPOT-5 HRG imagery , 2012 .

[89]  Sanja Fidler,et al.  Enhancing Road Maps by Parsing Aerial Images Around the World , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[90]  Gui-Song Xia,et al.  A Comparative Study of Sampling Analysis in the Scene Classification of Optical High-Spatial Resolution Remote Sensing Imagery , 2015, Remote. Sens..

[91]  Melba M. Crawford,et al.  Domain Adaptation With Preservation of Manifold Geometry for Hyperspectral Image Classification , 2016, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[92]  Josef Strobl,et al.  What’s wrong with pixels? Some recent developments interfacing remote sensing and GIS , 2001 .

[93]  Pierre Alliez,et al.  High-Resolution Semantic Labeling with Convolutional Neural Networks , 2016 .

[94]  Lorenzo Bruzzone,et al.  A Multilevel Context-Based System for Classification of Very High Spatial Resolution Images , 2006, IEEE Transactions on Geoscience and Remote Sensing.

[95]  William J. Emery,et al.  A neural network approach using multi-scale textural metrics from very high-resolution panchromatic imagery for urban land-use classification , 2009 .

[96]  Qian Song,et al.  Exploring the Use of Google Earth Imagery and Object-Based Methods in Land Use/Cover Mapping , 2013, Remote. Sens..

[97]  Pierre Alliez,et al.  Can semantic labeling methods generalize to any city? the inria aerial image labeling benchmark , 2017, 2017 IEEE International Geoscience and Remote Sensing Symposium (IGARSS).

[98]  R. Mathieu,et al.  Mapping private gardens in urban areas using object-oriented techniques and very high-resolution satellite imagery , 2007 .

[99]  U. Benz,et al.  Multi-resolution, object-oriented fuzzy analysis of remote sensing data for GIS-ready information , 2004 .

[100]  Jamie Sherrah,et al.  Effective semantic pixel labelling with convolutional networks and Conditional Random Fields , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[101]  Lawrence Carin,et al.  Multi-Task Learning for Classification with Dirichlet Process Priors , 2007, J. Mach. Learn. Res..

[102]  Yizhou Yu,et al.  Borrowing Treasures from the Wealthy: Deep Transfer Learning through Selective Joint Fine-Tuning , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[103]  C. Burnett,et al.  A multi-scale segmentation/object relationship modelling methodology for landscape analysis , 2003 .

[104]  Liangpei Zhang,et al.  A pixel shape index coupled with spectral information for classification of high spatial resolution remotely sensed imagery , 2006, IEEE Transactions on Geoscience and Remote Sensing.

[105]  Gui-Song Xia,et al.  Active learning for training sample selection in remote sensing image classification using spatial information , 2017 .

[106]  Julie Delon,et al.  Shape-based Invariant Texture Indexing , 2010, International Journal of Computer Vision.

[107]  Thomas Blaschke,et al.  Object based image analysis for remote sensing , 2010 .

[108]  Yang Shao,et al.  An evaluation of time-series smoothing algorithms for land-cover classifications using MODIS-NDVI multi-temporal data , 2016 .

[109]  Gui-Song Xia,et al.  Transferring Deep Convolutional Neural Networks for the Scene Classification of High-Resolution Remote Sensing Imagery , 2015, Remote. Sens..