How Well Do Deep Learning-Based Methods for Land Cover Classification and Object Detection Perform on High Resolution Remote Sensing Imagery?

© 2020 by the authors. Land cover information plays an important role in mapping ecological and environmental changes in Earth's diverse landscapes for ecosystem monitoring. Remote sensing data have been widely used for the study of land cover, enabling efficient mapping of changes of the Earth surface from Space. Although the availability of high-resolution remote sensing imagery increases significantly every year, traditional land cover analysis approaches based on pixel and object levels are not optimal. Recent advancement in deep learning has achieved remarkable success on image recognition field and has shown potential in high spatial resolution remote sensing applications, including classification and object detection. In this paper, a comprehensive review on land cover classification and object detection approaches using high resolution imagery is provided. Through two case studies, we demonstrated the applications of the state-of-the-art deep learning models to high spatial resolution remote sensing data for land cover classification and object detection and evaluated their performances against traditional approaches. For a land cover classification task, the deep-learning-based methods provide an end-to-end solution by using both spatial and spectral information. They have shown better performance than the traditional pixel-based method, especially for the categories of different vegetation. For an objective detection task, the deep-learning-based object detection method achieved more than 98% accuracy in a large area; its high accuracy and efficiency could relieve the burden of the traditional, labour-intensive method. However, considering the diversity of remote sensing data, more training datasets are required in order to improve the generalisation and the robustness of deep learning-based models.

[1]  Thomas Blaschke,et al.  Monitoring recovery after earthquakes through the integration of remote sensing, GIS, and ground observations: the case of L’Aquila (Italy) , 2016 .

[2]  Lei Zhang,et al.  Object-based approach to national land cover mapping using HJ satellite imagery , 2014 .

[3]  J. Rogan,et al.  Remote sensing technology for mapping and monitoring land-cover and land-use change , 2004 .

[4]  Sotiris B. Kotsiantis,et al.  Supervised Machine Learning: A Review of Classification Techniques , 2007, Informatica.

[5]  Antonio Torralba,et al.  LabelMe: A Database and Web-Based Tool for Image Annotation , 2008, International Journal of Computer Vision.

[6]  Qian Du,et al.  Multisource Remote Sensing Data Classification Based on Convolutional Neural Network , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[7]  Ye Zhang,et al.  A light and faster regional convolutional neural network for object detection in optical remote sensing images , 2018, ISPRS Journal of Photogrammetry and Remote Sensing.

[8]  Bo Huang,et al.  Urban land-use mapping using a deep convolutional neural network with high spatial resolution multispectral remote sensing imagery , 2018, Remote Sensing of Environment.

[9]  Adam Van Etten,et al.  SpaceNet: A Remote Sensing Dataset and Challenge Series , 2018, ArXiv.

[10]  Peijun Du,et al.  A review of supervised object-based land-cover image classification , 2017 .

[11]  Roberto Cipolla,et al.  SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  Stefan W. Maier,et al.  Comparing object-based and pixel-based classifications for mapping savannas , 2011, Int. J. Appl. Earth Obs. Geoinformation.

[13]  Ding Xiao-ying The Application of eCognition in Land Use Projects , 2005 .

[14]  Mariana Belgiu,et al.  Random forest in remote sensing: A review of applications and future directions , 2016 .

[15]  Lorenzo Bruzzone,et al.  Classification of hyperspectral remote sensing images with support vector machines , 2004, IEEE Transactions on Geoscience and Remote Sensing.

[16]  Xiaogang Wang,et al.  Pyramid Scene Parsing Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Peng Gong,et al.  Global land cover mapping using Earth observation satellite data: Recent progresses and challenges , 2015 .

[18]  Arno Schäpe,et al.  Multiresolution Segmentation : an optimization approach for high quality multi-scale image segmentation , 2000 .

[19]  Francesco Visin,et al.  A guide to convolution arithmetic for deep learning , 2016, ArXiv.

[20]  Bertrand Le Saux,et al.  Semantic Segmentation of Earth Observation Data Using Multimodal and Multi-scale Deep Networks , 2016, ACCV.

[21]  Lianru Gao,et al.  CNN-based Large Scale Landsat Image Classification , 2018, 2018 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC).

[22]  Ying Li,et al.  Spectral-Spatial Classification of Hyperspectral Imagery with 3D Convolutional Neural Network , 2017, Remote. Sens..

[23]  Wei Li,et al.  DeepUNet: A Deep Fully Convolutional Network for Pixel-Level Sea-Land Segmentation , 2017, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[24]  L. Carvalho,et al.  ANALYSIS OF RAPIDEYE'S RED EDGE BAND FOR IMAGE SEGMENTATION AND CLASSIFICATION , 2012 .

[25]  Erin F. Baerwald,et al.  Barotrauma is a significant cause of bat fatalities at wind turbines , 2008, Current Biology.

[26]  A. G. Wacker,et al.  Minimum Distance Classification in Remote Sensing , 1972 .

[27]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[28]  Rahul Ramachandran,et al.  Deep learning for phenomena-based classification of Earth science images , 2017 .

[29]  Sven Behnke,et al.  Evaluation of Pooling Operations in Convolutional Architectures for Object Recognition , 2010, ICANN.

[30]  Sebastian Ramos,et al.  The Cityscapes Dataset for Semantic Urban Scene Understanding , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[31]  Robert C. Weih,et al.  OBJECT-BASED CLASSIFICATION VS . PIXEL-BASED CLASSIFICATION : COMPARITIVE IMPORTANCE OF MULTI-RESOLUTION IMAGERY , 2010 .

[32]  Ronald Kemker,et al.  Algorithms for semantic segmentation of multispectral remote sensing imagery using deep learning , 2017, ISPRS Journal of Photogrammetry and Remote Sensing.

[33]  Liming Zhou,et al.  Impacts of wind farms on land surface temperature , 2012 .

[34]  Agnieszka Ścibior,et al.  Living in habitats affected by wind turbines may result in an increase in corticosterone levels in ground dwelling animals , 2018 .

[35]  Luís Pádua,et al.  UAS, sensors, and data processing in agroforestry: a review towards practical applications , 2017 .

[36]  Thomas R. Loveland,et al.  A review of large area monitoring of land cover change using Landsat data , 2012 .

[37]  Alexis J. Comber,et al.  Random forest classification of salt marsh vegetation habitats using quad-polarimetric airborne SAR, elevation and optical RS data , 2014 .

[38]  Rasim Latifovic,et al.  Accuracy assessment using sub-pixel fractional error matrices of global land cover products derived from satellite data , 2004 .

[39]  Jon Atli Benediktsson,et al.  Spectral–Spatial Classification of Multispectral Images Using Kernel Feature Space Representation , 2014, IEEE Geoscience and Remote Sensing Letters.

[40]  Xiaohui Xie,et al.  AnatomyNet: Deep learning for fast and fully automated whole‐volume segmentation of head and neck anatomy , 2018, Medical physics.

[41]  C. Brodley,et al.  Decision tree classification of land cover from remotely sensed data , 1997 .

[42]  K. Moffett,et al.  Remote Sens , 2015 .

[43]  Robert S. Leiken,et al.  A User’s Guide , 2011 .

[44]  Chen Zheng,et al.  Semantic Segmentation of Remote Sensing Imagery Using Object-Based Markov Random Field Model With Regional Penalties , 2015, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[45]  Shunping Xiao,et al.  Small Object Detection in Optical Remote Sensing Images via Modified Faster R-CNN , 2018 .

[46]  Jungho Im,et al.  Support vector machines in remote sensing: A review , 2011 .

[47]  Hong Zhang,et al.  Automatic Ship Detection Based on RetinaNet Using Multi-Resolution Gaofen-3 Imagery , 2019, Remote. Sens..

[48]  R. Clemen Combining forecasts: A review and annotated bibliography , 1989 .

[49]  D.M. Mount,et al.  An Efficient k-Means Clustering Algorithm: Analysis and Implementation , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[50]  Robert A. Schowengerdt CHAPTER 9 – Thematic Classification , 2007 .

[51]  Thomas Blaschke,et al.  Object based image analysis for remote sensing , 2010 .

[52]  Bo Du,et al.  Scene Classification via a Gradient Boosting Random Convolutional Network Framework , 2016, IEEE Transactions on Geoscience and Remote Sensing.

[53]  Michele Volpi,et al.  Semantic segmentation of urban scenes by learning local class interactions , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[54]  Giles M. Foody,et al.  A relative evaluation of multiclass image classification by support vector machines , 2004, IEEE Transactions on Geoscience and Remote Sensing.

[55]  Miao Zhang,et al.  Crop Mapping Using PROBA-V Time Series Data at the Yucheng and Hongxing Farm in China , 2016, Remote. Sens..

[56]  Ian D. Reid,et al.  RefineNet: Multi-path Refinement Networks for High-Resolution Semantic Segmentation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[57]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[58]  Fei Xia,et al.  Fast Detection of Airports on Remote Sensing Images with Single Shot MultiBox Detector , 2018 .

[59]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[60]  Adam Van Etten,et al.  Satellite Imagery Multiscale Rapid Detection with Windowed Networks , 2018, WACV.

[61]  Augustin Colette,et al.  Regional climate model simulations indicate limited climatic impacts by operational and planned European wind farms , 2014, Nature Communications.

[62]  Stephan Nebiker,et al.  Building Change Detection from Historical Aerial Photographs Using Dense Image Matching and Object-Based Image Analysis , 2014, Remote. Sens..

[63]  Sergey Ioffe,et al.  Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning , 2016, AAAI.

[64]  Tong Zhang,et al.  Deep Learning Based Feature Selection for Remote Sensing Scene Classification , 2015, IEEE Geoscience and Remote Sensing Letters.

[65]  O. Mutanga,et al.  Evaluating the impact of red-edge band from Rapideye image for classifying insect defoliation levels , 2014 .

[66]  Xiaojuan Qi,et al.  ICNet for Real-Time Semantic Segmentation on High-Resolution Images , 2017, ECCV.

[67]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[68]  A. Belward,et al.  GLC2000: a new approach to global land cover mapping from Earth observation data , 2005 .

[69]  Yan Zhou,et al.  Fast Automatic Airport Detection in Remote Sensing Images Using Convolutional Neural Networks , 2018, Remote. Sens..

[70]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[71]  Tao Zhou,et al.  The Observed Impacts of Wind Farms on Local Vegetation Growth in Northern China , 2017, Remote. Sens..

[72]  Francesca Bovolo,et al.  Semisupervised One-Class Support Vector Machines for Classification of Remote Sensing Data , 2010, IEEE Transactions on Geoscience and Remote Sensing.

[73]  Xinkai Wu,et al.  Car Detection from Low-Altitude UAV Imagery with the Faster R-CNN , 2017 .

[74]  Tao Lei,et al.  Robust Vehicle Detection in Aerial Images Based on Cascaded Convolutional Neural Networks , 2017, Sensors.

[75]  Feiping Nie,et al.  Learning a Mahalanobis distance metric for data clustering and classification , 2008, Pattern Recognit..

[76]  Pietro Perona,et al.  Microsoft COCO: Common Objects in Context , 2014, ECCV.

[77]  Bin Pan,et al.  Cascade Convolutional Neural Network Based on Transfer-Learning for Aircraft Detection on High-Resolution Remote Sensing Images , 2017, J. Sensors.

[78]  Paul M. Mather,et al.  An assessment of the effectiveness of decision tree methods for land cover classification , 2003 .

[79]  Geoffrey H. Ball,et al.  ISODATA, A NOVEL METHOD OF DATA ANALYSIS AND PATTERN CLASSIFICATION , 1965 .

[80]  Russell G. Congalton,et al.  A review of assessing the accuracy of classifications of remotely sensed data , 1991 .

[81]  U. Benz,et al.  Multi-resolution, object-oriented fuzzy analysis of remote sensing data for GIS-ready information , 2004 .

[82]  Daniel Rueckert,et al.  Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[83]  Michael E. Schaepman,et al.  Sentinels for science: potential of Sentinel-1, -2, and -3 missions for scientific observations of ocean, cryosphere, and land , 2012 .

[84]  Kil To Chong,et al.  Object Detection in Very High-Resolution Aerial Images Using One-Stage Densely Connected Feature Pyramid Network , 2018, Sensors.

[85]  Giles M. Foody,et al.  The use of small training sets containing mixed pixels for accurate hard image classification: Training on mixed spectral responses for classification by a SVM , 2006 .

[86]  Chao Tian,et al.  Dense Fusion Classmate Network for Land Cover Classification , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[87]  Ali Farhadi,et al.  YOLO9000: Better, Faster, Stronger , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[88]  J. S. Rawat,et al.  Monitoring land use/cover change using remote sensing and GIS techniques: A case study of Hawalbagh block, district Almora, Uttarakhand, India , 2015 .

[89]  Maggi Kelly,et al.  A spatial–temporal approach to monitoring forest disease spread using multi-temporal high spatial resolution imagery , 2006 .

[90]  Tao Li,et al.  HSI-CNN: A Novel Convolution Neural Network for Hyperspectral Image , 2018, 2018 International Conference on Audio, Language and Image Processing (ICALIP).

[91]  Howard D. Bondell,et al.  Minimum distance estimation for the logistic regression model , 2005 .

[92]  Jos B. T. M. Roerdink,et al.  The Watershed Transform: Definitions, Algorithms and Parallelization Strategies , 2000, Fundam. Informaticae.

[93]  Liu Pin,et al.  Aircraft Detection of High-Resolution Remote Sensing Image Based on Faster R-CNN Model and SSD Model , 2018, ICIGP 2018.

[94]  Geoff Smith,et al.  An evaluation of per-parcel land cover mapping using maximum likelihood class probabilities , 2003 .

[95]  Adam Van Etten,et al.  You Only Look Twice: Rapid Multi-Scale Object Detection In Satellite Imagery , 2018, ArXiv.

[96]  J. Strobl,et al.  Object-Oriented Image Processing in an Integrated GIS/Remote Sensing Environment and Perspectives for Environmental Applications , 2000 .

[97]  William L. Quinton,et al.  A decision-tree classification for low-lying complex land cover types within the zone of discontinuous permafrost , 2014 .

[98]  Seyed-Ahmad Ahmadi,et al.  V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation , 2016, 2016 Fourth International Conference on 3D Vision (3DV).

[99]  Xiaoxiao Li,et al.  Object-based land-cover classification for metropolitan Phoenix, Arizona, using aerial photography , 2014, Int. J. Appl. Earth Obs. Geoinformation.

[100]  M. Chevrel,et al.  The SPOT satellite remote sensing mission , 1981 .

[101]  Steven E. Franklin,et al.  A comparison of pixel-based and object-based image analysis with selected machine learning algorithms for the classification of agricultural landscapes using SPOT-5 HRG imagery , 2012 .

[102]  Xi Chen,et al.  A Feature-Based Approach of Decision Tree Classification to Map Time Series Urban Land Use and Land Cover with Landsat 5 TM and Landsat 8 OLI in a Coastal City, China , 2017, ISPRS Int. J. Geo Inf..

[103]  Lorenzo Bruzzone,et al.  Unsupervised retraining of a maximum likelihood classifier for the analysis of multitemporal remote sensing images , 2001, IEEE Trans. Geosci. Remote. Sens..

[104]  Martin Herold,et al.  Some challenges in global land cover mapping : An assessment of agreement and accuracy in existing 1 km datasets , 2008 .

[105]  Frank W. Gerlach,et al.  IKONOS satellite, imagery, and products , 2003 .

[106]  Jianhua Gong,et al.  UAV Remote Sensing for Urban Vegetation Mapping Using Random Forest and Texture Analysis , 2015, Remote. Sens..

[107]  Ali Farhadi,et al.  YOLOv3: An Incremental Improvement , 2018, ArXiv.

[108]  James C. Tilton,et al.  Image segmentation by region growing and spectral clustering with a natural convergence criterion , 1998, IGARSS '98. Sensing and Managing the Environment. 1998 IEEE International Geoscience and Remote Sensing. Symposium Proceedings. (Cat. No.98CH36174).

[109]  G. Meehl,et al.  The Importance of Land-Cover Change in Simulating Future Climates , 2005, Science.

[110]  Giorgos Mountrakis,et al.  A meta-analysis of remote sensing research on supervised pixel-based land-cover image classification processes: General guidelines for practitioners and future research , 2016 .

[111]  Shen Kuan-yu Wind energy resources and wind power generation in China , 2010 .

[112]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[113]  Rolf Adams,et al.  Seeded Region Growing , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[114]  Turgay Çelik,et al.  Unsupervised Change Detection in Satellite Images Using Principal Component Analysis and $k$-Means Clustering , 2009, IEEE Geoscience and Remote Sensing Letters.

[115]  C. Burnett,et al.  A multi-scale segmentation/object relationship modelling methodology for landscape analysis , 2003 .

[116]  Ali Farhadi,et al.  You Only Look Once: Unified, Real-Time Object Detection , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[117]  R. Congalton,et al.  Automated cropland mapping of continental Africa using Google Earth Engine cloud computing , 2017 .

[118]  S. Linden,et al.  The influence of urban structures on impervious surface maps from airborne hyperspectral data. , 2009 .

[119]  Russell G. Congalton,et al.  Global Land Cover Mapping: A Review and Uncertainty Analysis , 2014, Remote. Sens..

[120]  Nuno Vasconcelos,et al.  Cascade R-CNN: Delving Into High Quality Object Detection , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[121]  Li Fei-Fei,et al.  Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[122]  J. Townshend,et al.  Global land cover classi(cid:142) cation at 1 km spatial resolution using a classi(cid:142) cation tree approach , 2004 .

[123]  Wei Li,et al.  Major forest changes and land cover transitions based on plant functional types derived from the ESA CCI Land Cover product , 2016, Int. J. Appl. Earth Obs. Geoinformation.

[124]  Nandin-Erdene Tsendbazar,et al.  Global land cover mapping: current status and future trends , 2014 .

[125]  Luc Van Gool,et al.  The Pascal Visual Object Classes Challenge: A Retrospective , 2014, International Journal of Computer Vision.

[126]  Thomas Blaschke,et al.  Image Segmentation Methods for Object-based Analysis and Classification , 2004 .

[127]  S. Dwivedi,et al.  Obesity May Be Bad: Compressed Convolutional Networks for Biomedical Image Segmentation , 2020 .

[128]  Hitoshi Kiya,et al.  Super-Resolution Using Convolutional Neural Networks Without Any Checkerboard Artifacts , 2018, 2018 25th IEEE International Conference on Image Processing (ICIP).

[129]  Tao Fang,et al.  Selective convolutional neural networks and cascade classifiers for remote sensing image classification , 2017 .

[130]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[131]  Ross B. Girshick,et al.  Focal Loss for Dense Object Detection , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[132]  Yong Dou,et al.  Ship Detection Using Transfer Learned Single Shot Multi Box Detector , 2017 .

[133]  Jianzhou Wang,et al.  Intelligent optimized wind resource assessment and wind turbines selection in Huitengxile of Inner Mongolia, China , 2013 .

[134]  Wei Liu,et al.  SSD: Single Shot MultiBox Detector , 2015, ECCV.

[135]  Steffen Fritz,et al.  Identifying and quantifying uncertainty and spatial disagreement in the comparison of Global Land Cover for different applications , 2008 .

[136]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[137]  Liangpei Zhang,et al.  An Efficient and Robust Integrated Geospatial Object Detection Framework for High Spatial Resolution Remote Sensing Imagery , 2017, Remote. Sens..

[138]  Zhiguo Jiang,et al.  Ship detection in optical remote sensing images based on deep convolutional neural networks , 2017 .

[139]  Le Yu,et al.  Google Earth as a virtual globe tool for Earth science applications at the global scale: progress and perspectives , 2012 .

[140]  Alexandre Boulch,et al.  Deep learning for urban remote sensing , 2017, 2017 Joint Urban Remote Sensing Event (JURSE).

[141]  Jonathan D. Jordan,et al.  A Comparison of Fuzzy vs. Augmented-ISODATA Classification Algorithms for Cloud-Shadow Discrimination from Landsat Images , 2002 .