Cascaded Recurrent Neural Networks for Hyperspectral Image Classification

By considering the spectral signature as a sequence, recurrent neural networks (RNNs) have been successfully used to learn discriminative features from hyperspectral images (HSIs) recently. However, most of these models only input the whole spectral bands into RNNs directly, which may not fully explore the specific properties of HSIs. In this paper, we propose a cascaded RNN model using gated recurrent units to explore the redundant and complementary information of HSIs. It mainly consists of two RNN layers. The first RNN layer is used to eliminate redundant information between adjacent spectral bands, while the second RNN layer aims to learn the complementary information from nonadjacent spectral bands. To improve the discriminative ability of the learned features, we design two strategies for the proposed model. Besides, considering the rich spatial information contained in HSIs, we further extend the proposed model to its spectral–spatial counterpart by incorporating some convolutional layers. To test the effectiveness of our proposed models, we conduct experiments on two widely used HSIs. The experimental results show that our proposed models can achieve better results than the compared models.

[1]  Mariana Belgiu,et al.  Random forest in remote sensing: A review of applications and future directions , 2016 .

[2]  Tin Kam Ho,et al.  The Random Subspace Method for Constructing Decision Forests , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[3]  Lorenzo Bruzzone,et al.  Classification of hyperspectral remote sensing images with support vector machines , 2004, IEEE Transactions on Geoscience and Remote Sensing.

[4]  Nikolaos Doulamis,et al.  Deep supervised learning for hyperspectral data classification through convolutional neural networks , 2015, 2015 IEEE International Geoscience and Remote Sensing Symposium (IGARSS).

[5]  Qian Du,et al.  Local Binary Patterns and Extreme Learning Machine for Hyperspectral Imagery Classification , 2015, IEEE Transactions on Geoscience and Remote Sensing.

[6]  Lichao Mou,et al.  Learning a Transferable Change Rule from a Recurrent Neural Network for Land Cover Change Detection , 2016, Remote. Sens..

[7]  Jason Weston,et al.  Natural Language Processing (Almost) from Scratch , 2011, J. Mach. Learn. Res..

[8]  Xiao Xiang Zhu,et al.  Deep Learning in Remote Sensing: A Comprehensive Review and List of Resources , 2017, IEEE Geoscience and Remote Sensing Magazine.

[9]  Antonio J. Plaza,et al.  Sparse Unmixing-Based Change Detection for Multitemporal Hyperspectral Images , 2016, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[10]  Qian Du,et al.  Multisource Remote Sensing Data Classification Based on Convolutional Neural Network , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[11]  Yoshua Bengio,et al.  On the Properties of Neural Machine Translation: Encoder–Decoder Approaches , 2014, SSST@EMNLP.

[12]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  Melba M. Crawford,et al.  Manifold-Learning-Based Feature Extraction for Classification of Hyperspectral Data: A Review of Advances in Manifold Learning , 2014, IEEE Signal Processing Magazine.

[14]  J. A. Gualtieri,et al.  Support vector machines for classification of hyperspectral data , 2000, IGARSS 2000. IEEE 2000 International Geoscience and Remote Sensing Symposium. Taking the Pulse of the Planet: The Role of Remote Sensing in Managing the Environment. Proceedings (Cat. No.00CH37120).

[15]  Gui-Song Xia,et al.  Transferring Deep Convolutional Neural Networks for the Scene Classification of High-Resolution Remote Sensing Imagery , 2015, Remote. Sens..

[16]  Jonathan Cheung-Wai Chan,et al.  Learning and Transferring Deep Joint Spectral–Spatial Features for Hyperspectral Classification , 2017, IEEE Transactions on Geoscience and Remote Sensing.

[17]  Jürgen Schmidhuber,et al.  Framewise phoneme classification with bidirectional LSTM and other neural network architectures , 2005, Neural Networks.

[18]  Gui-Song Xia,et al.  Bag-of-Visual-Words Scene Classifier With Local and Global Features for High Spatial Resolution Remote Sensing Imagery , 2016, IEEE Geoscience and Remote Sensing Letters.

[19]  Bor-Chen Kuo,et al.  Feature Mining for Hyperspectral Image Classification , 2013, Proceedings of the IEEE.

[20]  Trevor Darrell,et al.  Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation , 2013, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[21]  Qingshan Liu,et al.  Integrating Convolutional Neural Network and Gated Recurrent Unit for Hyperspectral Image Spectral-Spatial Classification , 2018, PRCV.

[22]  Paul Rodríguez,et al.  A Recurrent Neural Network that Learns to Count , 1999, Connect. Sci..

[23]  Jon Atli Benediktsson,et al.  A Survey on Spectral–Spatial Classification Techniques Based on Attribute Profiles , 2015, IEEE Transactions on Geoscience and Remote Sensing.

[24]  Qian Du,et al.  Joint Within-Class Collaborative Representation for Hyperspectral Image Classification , 2014, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[25]  Antonio J. Plaza,et al.  This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination. IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING 1 Spectral–Spatial Classification of Hyperspectral Data Usi , 2022 .

[26]  Naoto Yokoya,et al.  Advances in Hyperspectral Image and Signal Processing: A Comprehensive Overview of the State of the Art , 2017, IEEE Geoscience and Remote Sensing Magazine.

[27]  Aleksandra Pizurica,et al.  Semisupervised Local Discriminant Analysis for Feature Extraction in Hyperspectral Images , 2013, IEEE Transactions on Geoscience and Remote Sensing.

[28]  Antonio J. Plaza,et al.  Robust Matrix Discriminative Analysis for Feature Extraction From Hyperspectral Images , 2017, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[29]  Lorenzo Bruzzone,et al.  Two-Stream Deep Architecture for Hyperspectral Image Classification , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[30]  Johannes R. Sveinsson,et al.  Automatic Spectral–Spatial Classification Framework Based on Attribute Profiles and Supervised Feature Extraction , 2014, IEEE Transactions on Geoscience and Remote Sensing.

[31]  Shutao Li,et al.  Learning to Diversify Deep Belief Networks for Hyperspectral Image Classification , 2017, IEEE Transactions on Geoscience and Remote Sensing.

[32]  Randolph L. Moses,et al.  Application of Model-Based Change Detection to Airborne VNIR/SWIR Hyperspectral Imagery , 2012, IEEE Transactions on Geoscience and Remote Sensing.

[33]  Qingshan Liu,et al.  Matrix-Based Discriminant Subspace Ensemble for Hyperspectral Image Spatial–Spectral Feature Fusion , 2016, IEEE Transactions on Geoscience and Remote Sensing.

[34]  Johannes R. Sveinsson,et al.  Spectral and spatial classification of hyperspectral data using SVMs and morphological profiles , 2008, 2007 IEEE International Geoscience and Remote Sensing Symposium.

[35]  Liangpei Zhang,et al.  Hyperspectral Image Classification by Nonlocal Joint Collaborative Representation With a Locally Adaptive Dictionary , 2014, IEEE Transactions on Geoscience and Remote Sensing.

[36]  Jun Li,et al.  Recent Advances on Spectral–Spatial Hyperspectral Image Classification: An Overview and New Guidelines , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[37]  Jon Atli Benediktsson,et al.  Nonlinear Multiple Kernel Learning With Multiple-Structure-Element Extended Morphological Profiles for Hyperspectral Image Classification , 2016, IEEE Transactions on Geoscience and Remote Sensing.

[38]  M. S. Moran,et al.  Opportunities and limitations for image-based remote sensing in precision crop management , 1997 .

[39]  Camille Couprie,et al.  Learning Hierarchical Features for Scene Labeling , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[40]  Xuelong Li,et al.  Scene Parsing From an MAP Perspective , 2015, IEEE Transactions on Cybernetics.

[41]  Yoshua Bengio,et al.  Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling , 2014, ArXiv.

[42]  Qingshan Liu,et al.  Bidirectional-Convolutional LSTM Based Spectral-Spatial Feature Learning for Hyperspectral Image Classification , 2017, Remote. Sens..

[43]  Bo Du,et al.  Slow Feature Analysis for Change Detection in Multispectral Imagery , 2014, IEEE Transactions on Geoscience and Remote Sensing.

[44]  Yansheng Li,et al.  Unsupervised Spectral–Spatial Feature Learning With Stacked Sparse Autoencoder for Hyperspectral Imagery Classification , 2015, IEEE Geoscience and Remote Sensing Letters.

[45]  Joydeep Ghosh,et al.  Investigation of the random forest framework for classification of hyperspectral data , 2005, IEEE Transactions on Geoscience and Remote Sensing.

[46]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[47]  Pedram Ghamisi,et al.  Spectral and Spatial Classification of Hyperspectral Data , 2015 .

[48]  Xiaoqiang Lu,et al.  Scene Recognition by Manifold Regularized Deep Learning Architecture , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[49]  Geoffrey E. Hinton,et al.  Speech recognition with deep recurrent neural networks , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[50]  Xiuping Jia,et al.  Deep Feature Extraction and Classification of Hyperspectral Images Based on Convolutional Neural Networks , 2016, IEEE Transactions on Geoscience and Remote Sensing.

[51]  Hao Wu,et al.  Convolutional Recurrent Neural Networks forHyperspectral Data Classification , 2017, Remote. Sens..

[52]  Ying Li,et al.  Spectral-Spatial Classification of Hyperspectral Imagery with 3D Convolutional Neural Network , 2017, Remote. Sens..

[53]  William J. Emery,et al.  Land Cover Mapping with Higher Order Graph-Based Co-Occurrence Model , 2018, Remote. Sens..

[54]  Jürgen Schmidhuber,et al.  Deep learning in neural networks: An overview , 2014, Neural Networks.

[55]  Qingshan Liu,et al.  Hyperspectral Image Classification Using Spectral-Spatial LSTMs , 2017, CCCV.

[56]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[57]  Quoc V. Le,et al.  Sequence to Sequence Learning with Neural Networks , 2014, NIPS.

[58]  Ronald J. Williams,et al.  A Learning Algorithm for Continually Running Fully Recurrent Neural Networks , 1989, Neural Computation.

[59]  Bo Du,et al.  Deep Learning for Remote Sensing Data: A Technical Tutorial on the State of the Art , 2016, IEEE Geoscience and Remote Sensing Magazine.

[60]  Xing Zhao,et al.  Spectral–Spatial Classification of Hyperspectral Data Based on Deep Belief Network , 2015, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[61]  Fang Tang,et al.  Deep Learning With Grouped Features for Spatial Spectral Classification of Hyperspectral Images , 2017, IEEE Geoscience and Remote Sensing Letters.

[62]  Antonio J. Plaza,et al.  A Discontinuity Preserving Relaxation Scheme for Spectral–Spatial Hyperspectral Image Classification , 2016, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[63]  Hermann Ney,et al.  From Feedforward to Recurrent LSTM Neural Networks for Language Modeling , 2015, IEEE/ACM Transactions on Audio, Speech, and Language Processing.

[64]  M. Bauer,et al.  Airborne hyperspectral remote sensing to assess spatial distribution of water quality characteristics in large rivers: the Mississippi River and its tributaries in Minnesota. , 2013 .

[65]  Xuelong Li,et al.  Semi-Supervised Multitask Learning for Scene Recognition , 2015, IEEE Transactions on Cybernetics.

[66]  Jon Atli Benediktsson,et al.  Spectral–Spatial Hyperspectral Image Classification via Multiscale Adaptive Sparse Representation , 2014, IEEE Transactions on Geoscience and Remote Sensing.

[67]  Francesca Bovolo,et al.  Updating Land-Cover Maps by Classification of Image Time Series: A Novel Change-Detection-Driven Transfer Learning Approach , 2013, IEEE Transactions on Geoscience and Remote Sensing.

[68]  J. Benediktsson,et al.  New Frontiers in Spectral-Spatial Hyperspectral Image Classification: The Latest Advances Based on Mathematical Morphology, Markov Random Fields, Segmentation, Sparse Representation, and Deep Learning , 2018, IEEE Geoscience and Remote Sensing Magazine.

[69]  Yunsong Li,et al.  Hyperspectral image reconstruction by deep convolutional neural network for classification , 2017, Pattern Recognit..

[70]  Razvan Pascanu,et al.  On the difficulty of training recurrent neural networks , 2012, ICML.

[71]  孙玉宝,et al.  Matrix-Based Discriminant Subspace Ensemble for Hyperspectral Image Spatial–Spectral Feature Fusion , 2015 .

[72]  William J. Emery,et al.  Object-Based Convolutional Neural Network for High-Resolution Imagery Classification , 2017, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[73]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[74]  G. F. Hughes,et al.  On the mean accuracy of statistical pattern recognizers , 1968, IEEE Trans. Inf. Theory.

[75]  Navdeep Jaitly,et al.  Towards End-To-End Speech Recognition with Recurrent Neural Networks , 2014, ICML.

[76]  Cheng Shi,et al.  Superpixel-based 3D deep neural networks for hyperspectral image classification , 2018, Pattern Recognit..

[77]  Jie Geng,et al.  Spectral–Spatial Classification of Hyperspectral Image Based on Deep Auto-Encoder , 2016, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[78]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[79]  Gang Wang,et al.  Deep Learning-Based Classification of Hyperspectral Data , 2014, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[80]  Trac D. Tran,et al.  Hyperspectral Image Classification Using Dictionary-Based Sparse Representation , 2011, IEEE Transactions on Geoscience and Remote Sensing.

[81]  Renlong Hang,et al.  Dimensionality Reduction of Hyperspectral Image Using Spatial Regularized Local Graph Discriminant Embedding , 2018, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[82]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.

[83]  Jitendra Malik,et al.  Region-Based Convolutional Networks for Accurate Object Detection and Segmentation , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[84]  Jitendra Malik,et al.  Recurrent Network Models for Human Dynamics , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[85]  Ben Somers,et al.  Heathland conservation status mapping through integration of hyperspectral mixture analysis and decision tree classifiers , 2012 .

[86]  Jungho Im,et al.  Support vector machines in remote sensing: A review , 2011 .

[87]  Wei Li,et al.  Diverse Region-Based CNN for Hyperspectral Image Classification , 2018, IEEE Transactions on Image Processing.

[88]  Fan Zhang,et al.  Deep Convolutional Neural Networks for Hyperspectral Image Classification , 2015, J. Sensors.