A Biologically Interpretable Two-Stage Deep Neural Network (BIT-DNN) for Vegetation Recognition From Hyperspectral Imagery

Spectral–spatial-based deep learning models have recently proven to be effective in hyper-spectral image (HSI) classification for various earth monitoring applications such as land cover classification and agricultural monitoring. However, due to the nature of “black-box” model representation, how to explain and interpret the learning process and the model decision, especially for vegetation classification, remains an open challenge. This study proposes a novel interpretable deep learning model—a biologically interpretable two-stage deep neural network (BIT-DNN), by incorporating the prior-knowledge (i.e., biophysical and biochemical attributes and their hierarchical structures of target entities)-based spectral–spatial feature transformation into the proposed framework, capable of achieving both high accuracy and interpretability on HSI-based classification tasks. The proposed model introduces a two-stage feature learning process: in the first stage, an enhanced interpretable feature block extracts the low-level spectral features associated with the biophysical and biochemical attributes of target entities; and in the second stage, an interpretable capsule block extracts and encapsulates the high-level joint spectral–spatial features representing the hierarchical structure of biophysical and biochemical attributes of these target entities, which provides the model an improved performance on classification and intrinsic interpretability with reduced computational complexity. We have tested and evaluated the model using four real HSI data sets for four separate tasks (i.e., plant species classification, land cover classification, urban scene recognition, and crop disease recognition tasks). The proposed model has been compared with five state-of-the-art deep learning models. The results demonstrate that the proposed model has competitive advantages in terms of both classification accuracy and model interpretability, especially for vegetation classification.

[1]  Gang Chen,et al.  Identification of Forested Landslides Using LiDar Data, Object-based Image Analysis, and Machine Learning Algorithms , 2015, Remote. Sens..

[2]  Xiaogang Wang,et al.  Orientation Invariant Feature Embedding and Spatial Temporal Regularization for Vehicle Re-identification , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[3]  Prasad S. Thenkabail,et al.  Spectral Bioindicators of Photosynthetic Efciency and Vegetation Stress , 2016 .

[4]  Ulas Bagci,et al.  Risk Stratification of Lung Nodules Using 3D CNN-Based Multi-task Learning , 2017, IPMI.

[5]  Nikolaos Doulamis,et al.  Deep supervised learning for hyperspectral data classification through convolutional neural networks , 2015, 2015 IEEE International Geoscience and Remote Sensing Symposium (IGARSS).

[6]  Jun Huang,et al.  Spectral-Spatial Attention Networks for Hyperspectral Image Classification , 2019, Remote. Sens..

[7]  Yan Xu,et al.  Deep learning of feature representation with multiple instance learning for medical image analysis , 2014, 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[8]  Ribana Roscher,et al.  Deep Self-taught Learning for Remote Sensing Image Classification , 2017, ArXiv.

[9]  Xiao Xiang Zhu,et al.  Deep Learning in Remote Sensing: A Comprehensive Review and List of Resources , 2017, IEEE Geoscience and Remote Sensing Magazine.

[10]  Jianxia Ma,et al.  Bi‐LSTM+CRF‐based named entity recognition in scientific papers in the field of ecological restoration technology , 2019 .

[11]  Jeffrey L. Gunsolus,et al.  Knowledge networks: an avenue to ecological management of invasive weeds , 2003, Weed Science.

[12]  G. Rondeaux,et al.  Optimization of soil-adjusted vegetation indices , 1996 .

[13]  Brandon M. Greenwell,et al.  Interpretable Machine Learning , 2019, Hands-On Machine Learning with R.

[14]  J. Zico Kolter,et al.  Gradient descent GAN optimization is locally stable , 2017, NIPS.

[15]  Qian Du,et al.  Feature-Driven Multilayer Visualization for Remotely Sensed Hyperspectral Imagery , 2010, IEEE Transactions on Geoscience and Remote Sensing.

[16]  Joachim Denzler,et al.  Deep learning and process understanding for data-driven Earth system science , 2019, Nature.

[17]  Jessica A. Faust,et al.  Imaging Spectroscopy and the Airborne Visible/Infrared Imaging Spectrometer (AVIRIS) , 1998 .

[18]  John T. Stasko,et al.  Toward a Deeper Understanding of the Role of Interaction in Information Visualization , 2007, IEEE Transactions on Visualization and Computer Graphics.

[19]  Michael S. Lew,et al.  Deep learning for visual understanding: A review , 2016, Neurocomputing.

[20]  S. Lange,et al.  [The kappa coefficient]. , 2007, Deutsche medizinische Wochenschrift.

[21]  Shanjun Mao,et al.  Spectral–spatial classification of hyperspectral images using deep convolutional neural networks , 2015 .

[22]  Baskar Ganapathysubramanian,et al.  An explainable deep machine vision framework for plant stress phenotyping , 2018, Proceedings of the National Academy of Sciences.

[23]  Germain Forestier,et al.  Knowledge-based region labeling for remote sensing image interpretation , 2012, Comput. Environ. Urban Syst..

[24]  Min Zuo,et al.  CropDeep: The Crop Vision Dataset for Deep-Learning-Based Classification and Detection in Precision Agriculture , 2019, Sensors.

[25]  Simon K. Warfield,et al.  Active Deep Learning with Fisher Information for Patch-Wise Semantic Segmentation , 2018, DLMIA/ML-CDS@MICCAI.

[26]  K. P. Soman,et al.  Dimensionally Reduced Features for Hyperspectral Image Classification Using Deep Learning , 2018, ICC 2018.

[27]  John R. Miller,et al.  Hyperspectral vegetation indices and novel algorithms for predicting green LAI of crop canopies: Modeling and validation in the context of precision agriculture , 2004 .

[28]  Timothy A. Warner,et al.  Implementation of machine-learning classification in remote sensing: an applied review , 2018 .

[29]  Thomas G. Dietterich.  Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms , 1998, Neural Computation.

[30]  Gui-Song Xia,et al.  Transferring Deep Convolutional Neural Networks for the Scene Classification of High-Resolution Remote Sensing Imagery , 2015, Remote. Sens..

[31]  S. K. McFeeters.  The use of the Normalized Difference Water Index (NDWI) in the delineation of open water features , 1996 .

[32]  Luis Alonso,et al.  Evaluation of Sentinel-2 Red-Edge Bands for Empirical Estimation of Green LAI and Chlorophyll Content , 2011, Sensors.

[33]  J. Peñuelas,et al.  The red edge position and shape as indicators of plant chlorophyll content, biomass and hydric status. , 1994 .

[34]  J. Gamon,et al.  The photochemical reflectance index: an optical indicator of photosynthetic radiation use efficiency across species, functional types, and nutrient levels , 1997, Oecologia.

[35]  Xiaoqiu Chen,et al.  Assessing plant senescence reflectance index-retrieved vegetation phenology and its spatiotemporal response to climate change in the Inner Mongolian Grassland , 2016, International Journal of Biometeorology.

[36]  Andreas Kamilaris,et al.  Deep learning in agriculture: A survey , 2018, Comput. Electron. Agric..

[37]  Jan G. P. W. Clevers,et al.  Remote estimation of crop and grass chlorophyll and nitrogen content using red-edge bands on Sentinel-2 and -3 , 2013, Int. J. Appl. Earth Obs. Geoinformation.

[38]  C. Smith.  Diagnostic tests (1) – sensitivity and specificity , 2012, Phlebology.

[39]  Jonathan Cheung-Wai Chan,et al.  Learning and Transferring Deep Joint Spectral–Spatial Features for Hyperspectral Classification , 2017, IEEE Transactions on Geoscience and Remote Sensing.

[40]  Shihong Du,et al.  Spectral–Spatial Feature Extraction for Hyperspectral Image Classification: A Dimension Reduction and Deep Learning Approach , 2016, IEEE Transactions on Geoscience and Remote Sensing.

[41]  Geoffrey E. Hinton,et al.  Dynamic Routing Between Capsules , 2017, NIPS.

[42]  Filiberto Pla,et al.  Capsule Networks for Hyperspectral Image Classification , 2019, IEEE Transactions on Geoscience and Remote Sensing.

[43]  Yang Wang,et al.  MARTA GANs: Unsupervised Representation Learning for Remote Sensing Image Classification , 2016, IEEE Geoscience and Remote Sensing Letters.

[44]  Richard Lepage,et al.  Knowledge-Based Image Understanding Systems: A Survey , 1997, Comput. Vis. Image Underst..

[45]  Deborshi Goswami.  Application of capsule networks for image classification on complex datasets , 2019 .

[46]  Carlos Guestrin,et al.  Model-Agnostic Interpretability of Machine Learning , 2016, ArXiv.

[47]  Honglak Lee,et al.  Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations , 2009, ICML '09.

[48]  R. Congalton,et al.  Accuracy assessment: a user's perspective , 1986 .

[49]  J. Cihlar,et al.  Relation between the normalized difference vegetation index and ecological variables , 1991 .

[50]  DaoYu Lin,et al.  Deep Unsupervised Representation Learning for Remote Sensing Images. , 2016 .

[51]  Zhiming Luo,et al.  Spectral–Spatial Residual Network for Hyperspectral Image Classification: A 3-D Deep Learning Framework , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[52]  Xiao Xiang Zhu,et al.  Fully conv-deconv network for unsupervised spectral-spatial feature extraction of hyperspectral imagery via residual learning , 2017, 2017 IEEE International Geoscience and Remote Sensing Symposium (IGARSS).

[53]  Jaegul Choo,et al.  Visual Analytics for Explainable Deep Learning , 2018, IEEE Computer Graphics and Applications.

[54]  Yi Lu,et al.  Auto-Encoder Based for High Spectral Dimensional Data Classification and Visualization , 2017, 2017 IEEE Second International Conference on Data Science in Cyberspace (DSC).

[55]  Roberta E. Martin,et al.  Leaf aging of Amazonian canopy trees as revealed by spectral and physiochemical measurements. , 2017, The New phytologist.

[56]  Yang Gao,et al.  Dimensionality Reduction for Hyperspectral Data Based on Class-Aware Tensor Neighborhood Graph and Patch Alignment , 2015, IEEE Transactions on Neural Networks and Learning Systems.