Landslide Image Captioning Method Based on Semantic Gate and Bi-Temporal LSTM

When a landslide happens, it is important to recognize the hazard-affected bodies surrounding the landslide for the risk assessment and emergency rescue. In order to realize the recognition, the spatial relationship between landslides and other geographic objects such as residence, roads and schools needs to be defined. Comparing with semantic segmentation and instance segmentation that can only recognize the geographic objects separately, image captioning can provide richer semantic information including the spatial relationship among these objects. However, the traditional image captioning methods based on RNNs have two main shortcomings: the errors in the prediction process are often accumulated and the location of attention is not always accurate which would lead to misjudgment of risk. To handle these problems, a landslide image interpretation network based on a semantic gate and a bi-temporal long-short term memory network (SG-BiTLSTM) is proposed in this paper. In the SG-BiTLSTM architecture, a U-Net is employed as an encoder to extract features of the images and generate the mask maps of the landslides and other geographic objects. The decoder of this structure consists of two interactive long-short term memory networks (LSTMs) to describe the spatial relationship among these geographic objects so that to further determine the role of the classified geographic objects for identifying the hazard-affected bodies. The purpose of this research is to judge the hazard-affected bodies of the landslide (i.e., buildings and roads) through the SG-BiTLSTM network to provide geographic information support for emergency service. The remote sensing data was taken by Worldview satellite after the Wenchuan earthquake happened in 2008. The experimental results demonstrate that SG-BiTLSTM network shows remarkable improvements on the recognition of landslide and hazard-affected bodies, compared with the traditional LSTM (the Baseline Model), the BLEU1 of the SG-BiTLSTM is improved by 5.89%, the matching rate between the mask maps and the focus matrix of the attention is improved by 42.81%. In conclusion, the SG-BiTLSTM network can recognize landslides and the hazard-affected bodies simultaneously to provide basic geographic information service for emergency decision-making.

[1]  Qing Zhou,et al.  Planet Image-Based Inventorying and Machine Learning-Based Susceptibility Mapping for the Landslides Triggered by the 2018 Mw6.6 Tomakomai, Japan Earthquake , 2019, Remote. Sens..

[2]  Suzana Dragićević,et al.  Analyzing the Effects of Temporal Resolution and Classification Confidence for Modeling Land Cover Change with Long Short-Term Memory Networks , 2019, Remote. Sens..

[3]  Fei Wang,et al.  Multi-Scale Semantic Segmentation and Spatial Relationship Recognition of Remote Sensing Images Based on an Attention Model , 2019, Remote. Sens..

[4]  Bin Luo,et al.  Bidirectional Convolutional LSTM Neural Network for Remote Sensing Image Super-Resolution , 2019, Remote. Sens..

[5]  Jiang Bian,et al.  Simplified Evaluation of Cotton Water Stress Using High Resolution Unmanned Aerial Vehicle Thermal Imagery , 2019, Remote. Sens..

[6]  Saro Lee,et al.  Landslide Susceptibility Mapping and Comparison Using Decision Tree Models: A Case Study of Jumunjin Area, Korea , 2018, Remote. Sens..

[7]  Emile Ndikumana,et al.  Deep Recurrent Neural Network for Agricultural Classification using multitemporal SAR Sentinel-1 for Camargue, France , 2018, Remote. Sens..

[8]  Thomas Blaschke,et al.  A Novel Ensemble Approach for Landslide Susceptibility Mapping (LSM) in Darjeeling and Kalimpong Districts, West Bengal, India , 2019, Remote. Sens..

[9]  Andrea Manconi,et al.  Mapping Landslides on EO Data: Performance of Deep Learning Models vs. Traditional Machine Learning Models , 2020, Remote. Sens..

[10]  Hod Lipson,et al.  Quantitative Phenotyping of Northern Leaf Blight in UAV Images Using Deep Learning , 2019, Remote. Sens..

[11]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  Biao Wang,et al.  Building Extraction in Very High Resolution Imagery by Dense-Attention Networks , 2018, Remote. Sens..

[13]  G. J. Hay,et al.  A multiscale framework for landscape analysis: Object-specific analysis and upscaling , 2001, Landscape Ecology.

[14]  Haihong Zhu,et al.  A Multiple-Feature Reuse Network to Extract Buildings from Remote Sensing Imagery , 2018, Remote. Sens..

[15]  Zhenwei Shi,et al.  Can a Machine Generate Humanlike Language Descriptions for a Remote Sensing Image? , 2017, IEEE Transactions on Geoscience and Remote Sensing.

[16]  Pengqiang Zhang,et al.  Spectral-spatial classification of hyperspectral imagery based on recurrent neural networks , 2018, Remote Sensing Letters.

[17]  Qi Zhou,et al.  Application of a parallel spectral–spatial convolution neural network in object-oriented remote sensing land use classification , 2018 .

[18]  Rasmus Nyholm Jørgensen,et al.  A Novel Spatio-Temporal FCN-LSTM Network for Recognizing Various Crop Types Using Multi-Temporal Radar Images , 2019, Remote. Sens..

[19]  Jie Dou,et al.  Evaluating GIS-Based Multiple Statistical Models and Data Mining for Earthquake and Rainfall-Induced Landslide Susceptibility Using the LiDAR DEM , 2019, Remote. Sens..

[20]  Feng Li,et al.  Fusion of Multiscale Convolutional Neural Networks for Building Extraction in Very High-Resolution Images , 2019, Remote. Sens..

[21]  Robert B. Fisher,et al.  Semantic Segmentation of SLAR Imagery with Convolutional LSTM Selectional AutoEncoders , 2019, Remote. Sens..

[22]  J. Gao,et al.  Identification and estimation of landslide-debris flow disaster risk in primary and middle school campuses in a mountainous area of Southwest China , 2017 .

[23]  Ning Jin,et al.  Using Neural Network to Identify the Severity of Wheat Fusarium Head Blight in the Field Environment , 2019, Remote. Sens..

[24]  Jiancheng Luo,et al.  Semantic segmentation of high spatial resolution images with deep neural networks , 2019, GIScience & Remote Sensing.

[25]  Xiang Zhao,et al.  Long Time Series Land Cover Classification in China from 1982 to 2015 Based on Bi-LSTM Deep Learning , 2019, Remote. Sens..

[26]  Xiangtao Zheng,et al.  Exploring Models and Data for Remote Sensing Image Caption Generation , 2017, IEEE Transactions on Geoscience and Remote Sensing.

[27]  An Chen,et al.  Regional disaster risk evaluation of China based on the universal risk model , 2017, Natural Hazards.

[28]  Saro Lee,et al.  Application of Ensemble-Based Machine Learning Models to Landslide Susceptibility Mapping , 2018, Remote. Sens..

[29]  Yi Cen,et al.  A Spectral-Spatial Cascaded 3D Convolutional Neural Network with a Convolutional Long Short-Term Memory Network for Hyperspectral Image Classification , 2019, Remote. Sens..

[30]  Weijia Li,et al.  Large-Scale Oil Palm Tree Detection from High-Resolution Satellite Images Using Two-Stage Convolutional Neural Networks , 2018, Remote. Sens..

[31]  Xiangtao Zheng,et al.  Semantic Descriptions of High-Resolution Remote Sensing Images , 2019, IEEE Geoscience and Remote Sensing Letters.

[32]  Anthony M. Filippi,et al.  Hyperspectral Image Classification Using Similarity Measurements-Based Deep Recurrent Neural Networks , 2019, Remote. Sens..

[33]  Geoffrey J. Hay,et al.  Geographic object-based image analysis (GEOBIA): emerging trends and future opportunities , 2018 .

[34]  Yu Zhu,et al.  Refinement of Landslide Susceptibility Map Using Persistent Scatterer Interferometry in Areas of Intense Mining Activities in the Karst Region of Southwest China , 2019, Remote. Sens..

[35]  Laura Longoni,et al.  Remote Sensing for Landslide Investigations: An Overview of Recent Achievements and Perspectives , 2014, Remote. Sens..

[36]  Xin Wang,et al.  Description Generation for Remote Sensing Images Using Attribute Attention Mechanism , 2019, Remote. Sens..

[37]  Qingshan Liu,et al.  Bidirectional-Convolutional LSTM Based Spectral-Spatial Feature Learning for Hyperspectral Image Classification , 2017, Remote. Sens..

[38]  Dongming Lu,et al.  Mapping Impervious Surfaces in Town-Rural Transition Belts Using China's GF-2 Imagery and Object-Based Deep CNNs , 2019, Remote. Sens..

[39]  Michael Ying Yang,et al.  Application of Deep Learning for Delineation of Visible Cadastral Boundaries from Remote Sensing Imagery , 2019, Remote. Sens..

[40]  Naser El-Sheimy,et al.  Using an Unmanned Aerial Vehicle-Based Digital Imaging System to Derive a 3D Point Cloud for Landslide Scarp Recognition , 2016, Remote. Sens..

[41]  Omid Ghorbanzadeh,et al.  UAV-Based Slope Failure Detection Using Deep-Learning Convolutional Neural Networks , 2019, Remote. Sens..

[42]  Gaohuan Liu,et al.  Evaluation and Comparison of Random Forest and A-LSTM Networks for Large-scale Winter Wheat Identification , 2019, Remote. Sens..

[43]  Jie Yang,et al.  Altimeter Observation-Based Eddy Nowcasting Using an Improved Conv-LSTM Network , 2019, Remote. Sens..

[44]  Zaiwu Gong,et al.  A review of risk analysis methods for natural disasters , 2019, Natural Hazards.

[45]  Gang Chen,et al.  Article in Press G Model International Journal of Applied Earth Observation and Geoinformation a Geobia Framework to Estimate Forest Parameters from Lidar Transects, Quickbird Imagery and Machine Learning: a Case Study in Quebec, Canada , 2022 .

[46]  Zhen Li,et al.  Examining the sensitivity of spatial scale in cellular automata Markov chain simulation of land use change , 2019, Int. J. Geogr. Inf. Sci..

[47]  Hao Wu,et al.  Convolutional Recurrent Neural Networks forHyperspectral Data Classification , 2017, Remote. Sens..

[48]  Omid Ghorbanzadeh,et al.  Landslide Detection Using Multi-Scale Image Segmentation and Different Machine Learning Models in the Higher Himalayas , 2019, Remote. Sens..