Landscape Classification with Deep Neural Networks

The application of deep learning, specifically deep convolutional neural networks (DCNNs), to the classification of remotely-sensed imagery of natural landscapes has the potential to greatly assist in the analysis and interpretation of geomorphic processes. However, the general usefulness of deep learning applied to conventional photographic imagery at a landscape scale is, at yet, largely unproven. If DCNN-based image classification is to gain wider application and acceptance within the geoscience community, demonstrable successes need to be coupled with accessible tools to retrain deep neural networks to discriminate landforms and land uses in landscape imagery. Here, we present an efficient approach to train/apply DCNNs with/on sets of photographic images, using a powerful graphical method called a conditional random field (CRF), to generate DCNN training and testing data using minimal manual supervision. We apply the method to several sets of images of natural landscapes, acquired from satellites, aircraft, unmanned aerial vehicles, and fixed camera installations. We synthesize our findings to examine the general effectiveness of transfer learning to landscape-scale image classification. Finally, we show how DCNN predictions on small regions of images might be used in conjunction with a CRF for highly accurate pixel-level classification of images.

[1]  Timothy J. Randle,et al.  Large-scale dam removal on the Elwha River, Washington, USA: River channel and floodplain geomorphic change , 2014 .

[2]  Gang Fu,et al.  Classification for High Resolution Remote Sensing Imagery Using a Fully Convolutional Network , 2017, Remote. Sens..

[3]  Michele Volpi,et al.  Dense Semantic Labeling of Subdecimeter Resolution Images With Convolutional Neural Networks , 2016, IEEE Transactions on Geoscience and Remote Sensing.

[4]  Nir Friedman,et al.  Probabilistic Graphical Models - Principles and Techniques , 2009 .

[5]  Uwe Stilla,et al.  Deep Learning Earth Observation Classification Using ImageNet Pretrained Networks , 2016, IEEE Geoscience and Remote Sensing Letters.

[6]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[7]  V. L. Mulder,et al.  The use of remote sensing in soil and terrain mapping — A review , 2011 .

[8]  R. Holman,et al.  The history and technical capabilities of Argus , 2007 .

[9]  Peijun Du,et al.  A review of supervised object-based land-cover image classification , 2017 .

[10]  J. Warrick,et al.  New Techniques to Measure Cliff Change from Historical Oblique Aerial Photographs and Structure-from-Motion Photogrammetry , 2016, Journal of Coastal Research.

[11]  Mark A. Fonstad,et al.  Topographic structure from motion: a new development in photogrammetric measurement , 2013 .

[12]  Gonzalo Pajares,et al.  Overview and Current Status of Remote Sensing Applications Based on Unmanned Aerial Vehicles (UAVs) , 2015 .

[13]  Mitchell D. Harley,et al.  UAVs for coastal surveying , 2016 .

[14]  W. Bertoldi,et al.  Assessment of morphological changes induced by flow and flood pulses in a gravel bed braided river: The Tagliamento River (Italy) , 2010 .

[15]  A. Tamminga,et al.  Hyperspatial Remote Sensing of Channel Reach Morphology and Hydraulic Fish Habitat Using an Unmanned Aerial Vehicle (UAV): A First Assessment in the Context of River Research and Management , 2015 .

[16]  Andrew McCallum,et al.  An Introduction to Conditional Random Fields for Relational Learning , 2007 .

[17]  Vladlen Koltun,et al.  Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials , 2011, NIPS.

[18]  Iasonas Kokkinos,et al.  DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[19]  Luisa Verdoliva,et al.  Land Use Classification in Remote Sensing Images by Convolutional Neural Networks , 2015, ArXiv.

[20]  Mark Sandler,et al.  MobileNetV2: Inverted Residuals and Linear Bottlenecks , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[21]  Amy Loutfi,et al.  Classification and Segmentation of Satellite Orthoimagery Using Convolutional Neural Networks , 2016, Remote. Sens..

[22]  Edward H. Adelson,et al.  Learning Gaussian Conditional Random Fields for Low-Level Vision , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[23]  James C. Gibeaut,et al.  Using UAS Hyperspatial RGB Imagery for Identifying Beach Zones along the South Texas Coast , 2017, Remote. Sens..

[24]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[25]  Sergey Ioffe,et al.  Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning , 2016, AAAI.

[26]  Mariana Belgiu,et al.  Random forest in remote sensing: A review of applications and future directions , 2016 .

[27]  Bram van Ginneken,et al.  A survey on deep learning in medical image analysis , 2017, Medical Image Anal..

[28]  Robbie Austrums,et al.  Subaerial gravel size measurement using topographic data derived from a UAV‐SfM approach , 2017 .

[29]  Vijay Vasudevan,et al.  Learning Transferable Architectures for Scalable Image Recognition , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[30]  Chao Liu,et al.  Cultivated land information extraction in UAV imagery based on deep convolutional neural network and transfer learning , 2017, Journal of Mountain Science.

[31]  Leon F. Palafox,et al.  Automated detection of geological landforms on Mars using Convolutional Neural Networks , 2017, Comput. Geosci..

[32]  Xiaoqiang Lu,et al.  Remote Sensing Image Scene Classification: Benchmark and State of the Art , 2017, Proceedings of the IEEE.

[33]  Martial Hebert,et al.  Discriminative Random Fields , 2006, International Journal of Computer Vision.

[34]  Fedor Baart,et al.  An Automated Method for Semantic Classification of Regions in Coastal Images , 2015 .

[35]  J. Brasington,et al.  Modeling the topography of shallow braided rivers using Structure-from-Motion photogrammetry , 2014 .

[36]  Sergey Ioffe,et al.  Rethinking the Inception Architecture for Computer Vision , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  H. Piégay,et al.  A new methodology for monitoring wood fluxes in rivers using a ground camera: Potential and limits , 2017 .

[38]  Jianfei Cai,et al.  Beyond pixels: A comprehensive survey from bottom-up to semantic image segmentation and cosegmentation , 2015, J. Vis. Commun. Image Represent..

[39]  José García Rodríguez,et al.  A Review on Deep Learning Techniques Applied to Semantic Segmentation , 2017, ArXiv.

[40]  Robert G. Bryant,et al.  Quantifying geomorphic and riparian land cover changes either side of a large flood event using airborne remote sensing: River Tay, Scotland , 1999 .

[41]  Fei-Fei Li,et al.  ImageNet: A large-scale hierarchical image database , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[42]  Andrew McCallum,et al.  Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data , 2001, ICML.

[43]  Gui-Song Xia,et al.  Transferring Deep Convolutional Neural Networks for the Scene Classification of High-Resolution Remote Sensing Imagery , 2015, Remote. Sens..

[44]  Daniel Conley,et al.  Video-based nearshore bathymetry estimation in macro-tidal environments , 2016 .

[45]  Pierre Alliez,et al.  Convolutional Neural Networks for Large-Scale Remote-Sensing Image Classification , 2017, IEEE Transactions on Geoscience and Remote Sensing.

[46]  Yichun Xie,et al.  Remote sensing imagery in vegetation mapping: a review , 2008 .

[47]  Xiaoxiao Li,et al.  Object-Based Land-Cover Mapping with High Resolution Aerial Photography at a County Scale in Midwestern USA , 2014, Remote. Sens..

[48]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[49]  Michael A. Wulder,et al.  Remote sensing methods in medium spatial resolution satellite data land cover classification of large areas , 2002 .

[50]  Bo Chen,et al.  MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications , 2017, ArXiv.

[51]  David P. Remsen,et al.  UAS-SfM for Coastal Research: Geomorphic Feature Extraction and Land Cover Classification from High-Resolution Elevation and Optical Imagery , 2017, Remote Sensing.

[52]  Jonathan A. Warrick,et al.  Large-scale dam removal on the Elwha River, Washington, USA: Source-to-sink sediment budget and synthesis , 2015 .

[53]  Patrice E. Carbonneau,et al.  Robotic photosieving from low‐cost multirotor sUAS: a proof‐of‐concept , 2018 .

[54]  Yi Yang,et al.  Attention to Scale: Scale-Aware Semantic Image Segmentation , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[55]  Mike J. Smith,et al.  Cameras and settings for aerial surveys in the geosciences , 2017 .

[56]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[57]  C. Hugenholtz,et al.  Geomorphological mapping with a small unmanned aircraft system (sUAS): Feature detection and accuracy assessment of a photogrammetrically-derived digital terrain model , 2013 .

[58]  Surya Ganguli,et al.  Identifying and attacking the saddle point problem in high-dimensional non-convex optimization , 2014, NIPS.

[59]  Martín Abadi,et al.  TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems , 2016, ArXiv.

[60]  Chad Hendrix,et al.  A Comparison of Urban Mapping Methods Using High-Resolution Digital Imagery , 2003 .

[61]  A. Lejay,et al.  On the use of the Radon transform to estimate longshore currents from video imagery , 2014 .

[62]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[63]  Francesco Mancini,et al.  Image classification methods applied to shoreline extraction on very high-resolution multispectral imagery , 2014 .

[64]  Andrew McCallum,et al.  Introduction to Statistical Relational Learning , 2007 .

[65]  Colin Pain,et al.  Applications of remote sensing in geomorphology , 2009 .

[66]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[67]  A. St‐Hilaire,et al.  Spatial distribution of thermal refuges analysed in relation to riverscape hydromorphology using airborne thermal infrared imagery , 2015 .

[68]  O. Mutanga,et al.  Multispectral and hyperspectral remote sensing for identification and mapping of wetland vegetation: a review , 2010, Wetlands Ecology and Management.