Land Cover Classification from Multi-temporal, Multi-spectral Remotely Sensed Imagery using Patch-Based Recurrent Neural Networks

Environmental sustainability research is dependent on accurate land cover information. Even with the increased number of satellite systems and sensors acquiring data with improved spectral, spatial, radiometric and temporal characteristics and the new data distribution policy, most existing land cover datasets are derived from a pixel-based, single-date multi-spectral remotely sensed image with an unacceptable accuracy. One major bottleneck for accuracy improvement is how to develop an accurate and effective image classification protocol. By incorporating and utilizing multi-spectral, multi-temporal and spatial information in remote sensing images and considering the inherit spatial and sequential interdependence among neighboring pixels, we propose a new patch-based recurrent neural network (PB-RNN) system tailored for classifying multi-temporal remote sensing data. The system is designed by incorporating distinctive characteristics of multi-temporal remote sensing data. In particular, it uses multi-temporal-spectral-spatial samples and deals with pixels contaminated by clouds/shadow present in multi-temporal data series. Using a Florida Everglades ecosystem study site covering an area of 771 square kilometers, the proposed PB-RNN system has achieved a significant improvement in the classification accuracy over a pixel-based recurrent neural network (RNN) system, a pixel-based single-image neural network (NN) system, a pixel-based multi-image NN system, a patch-based single-image NN system, and a patch-based multi-image NN system. For example, the proposed system achieves 97.21% classification accuracy while the pixel-based single-image NN system achieves 64.74%. By utilizing methods like the proposed PB-RNN one, we believe that much more accurate land cover datasets can be produced over large areas.

[1]  Matthew R. Walter,et al.  Listen, Attend, and Walk: Neural Mapping of Navigational Instructions to Action Sequences , 2015, AAAI.

[2]  Trevor Darrell,et al.  Long-term recurrent convolutional networks for visual recognition and description , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Danny Lo Seen,et al.  Crop area mapping in West Africa using landscape stratification of MODIS time series and comparison with existing global land products , 2012, Int. J. Appl. Earth Obs. Geoinformation.

[4]  R. Leemans.  Changes in Land use and land cover: A global perspective , 1995 .

[5]  Martín Abadi,et al.  TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems , 2016, ArXiv.

[6]  Russell G. Congalton,et al.  A review of assessing the accuracy of classifications of remotely sensed data , 1991 .

[7]  Lukás Burget,et al.  Recurrent neural network based language model , 2010, INTERSPEECH.

[8]  Xiaojun Yang.  Use of Archival Landsat Imagery to Monitor Urban Spatial Growth , 2011 .

[9]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[10]  Ting Liu,et al.  Document Modeling with Gated Recurrent Neural Network for Sentiment Classification , 2015, EMNLP.

[11]  Tara N. Sainath,et al.  Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups , 2012, IEEE Signal Processing Magazine.

[12]  Stuart E. Marsh,et al.  Cover- and density-based vegetation classifications of the Sonoran Desert using Landsat TM and ERS-1 SAR imagery , 2004 .

[13]  S. Carpenter,et al.  Global Consequences of Land Use , 2005, Science.

[14]  Geoffrey E. Hinton,et al.  Reducing the Dimensionality of Data with Neural Networks , 2006, Science.

[15]  Christopher A. Barnes,et al.  Completion of the 2006 National Land Cover Database for the conterminous United States. , 2011 .

[16]  James R. Anderson,et al.  A land use and land cover classification system for use with remote sensor data , 1976 .

[17]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[18]  Xiaojun Yang,et al.  Parameterizing Support Vector Machines for Land Cover Classification , 2011 .

[19]  Di Shi,et al.  An Assessment of Algorithmic Parameters Affecting Image Classification Accuracy by Random Forests , 2016 .

[20]  Suming Jin,et al.  A comprehensive change detection method for updating the National Land Cover Database to circa 2011 , 2013 .

[21]  Lichao Mou,et al.  Learning a Transferable Change Rule from a Recurrent Neural Network for Land Cover Change Detection , 2016, Remote. Sens..

[22]  Alex Graves,et al.  Neural Turing Machines , 2014, ArXiv.

[23]  A. Belward,et al.  GLC2000: a new approach to global land cover mapping from Earth observation data , 2005 .

[24]  Brian W. Barrett,et al.  Temporal optimisation of image acquisition for land cover classification with Random Forest and MODIS time-series , 2015, Int. J. Appl. Earth Obs. Geoinformation.

[25]  J. Wickham,et al.  Completion of the 2001 National Land Cover Database for the conterminous United States , 2007 .

[26]  Lukás Burget,et al.  Extensions of recurrent neural network language model , 2011, 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[27]  Christian W. Omlin,et al.  Credit Card Transactions, Fraud Detection, and Machine Learning: Modelling Time with LSTM Recurrent Neural Networks , 2009, Innovations in Neural Information Paradigms and Applications.

[28]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[29]  Bo Qu,et al.  Deep semantic understanding of high resolution remote sensing image , 2016, 2016 International Conference on Computer, Information and Telecommunication Systems (CITS).

[30]  Geoffrey E. Hinton,et al.  Speech recognition with deep recurrent neural networks , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[31]  I. Kanellopoulos,et al.  Strategies and best practice for neural network image classification , 1997 .

[32]  Damian Bargiel,et al.  Multi-Temporal Land-Cover Classification of Agricultural Areas in Two European Regions with High Resolution Spotlight TerraSAR-X Data , 2011, Remote. Sens..

[33]  Hankui K. Zhang,et al.  Finer resolution observation and monitoring of global land cover: first mapping results with Landsat TM and ETM+ data , 2013 .

[34]  Alex Graves,et al.  DRAW: A Recurrent Neural Network For Image Generation , 2015, ICML.

[35]  Fei-Fei Li,et al.  Deep visual-semantic alignments for generating image descriptions , 2015, CVPR.

[36]  C. Lo,et al.  The Influence of Geographic Sampling Methods on Vegetation Map Accuracy Evaluation in a Swampy Environment , 1998 .

[37]  Phil Blunsom,et al.  Teaching Machines to Read and Comprehend , 2015, NIPS.

[38]  Zhe Zhu,et al.  Object-based cloud and cloud shadow detection in Landsat imagery , 2012 .

[39]  Samy Bengio,et al.  Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks , 2015, NIPS.

[40]  Christoph Schneider,et al.  Spatio‐temporal prediction of snow cover in the Black Forest mountain range using remote sensing and a recurrent neural network , 2010 .

[41]  Phil Blunsom,et al.  Recurrent Continuous Translation Models , 2013, EMNLP.

[42]  Limin Yang,et al.  COMPLETION OF THE 1990S NATIONAL LAND COVER DATA SET FOR THE CONTERMINOUS UNITED STATES FROM LANDSAT THEMATIC MAPPER DATA AND ANCILLARY DATA SOURCES , 2001 .

[43]  Paul M. Mather,et al.  The use of backpropagating artificial neural networks in land cover classification , 2003 .

[44]  Nitish Srivastava,et al.  Unsupervised Learning of Video Representations using LSTMs , 2015, ICML.

[45]  John R. Jensen,et al.  Introductory Digital Image Processing: A Remote Sensing Perspective , 1986 .

[46]  Steven M. Davis,et al.  Everglades. The Ecosystem and Its Restoration , 1995 .

[47]  Lorenzo Bruzzone,et al.  A neural-statistical approach to multitemporal and multisource remote-sensing image classification , 1999, IEEE Trans. Geosci. Remote. Sens..

[48]  Hugo Carrão,et al.  Contribution of multispectral and multitemporal information from MODIS images to land cover classification , 2008 .

[49]  Ronan Collobert,et al.  Recurrent Convolutional Neural Networks for Scene Labeling , 2014, ICML.

[50]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[51]  Juan J. Flores,et al.  The application of artificial neural networks to the analysis of remotely sensed data , 2008 .

[52]  Xiuwen Liu,et al.  A patch-based convolutional neural network for remote sensing image classification , 2017, Neural Networks.

[53]  Geoffrey Zweig,et al.  Recent advances in deep learning for speech research at Microsoft , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.