Destruction from sky: Weakly supervised approach for destruction detection in satellite imagery

Abstract Natural and man-made disasters cause huge damage to built infrastructures and results in loss of human lives. The rehabilitation efforts and rescue operations are hampered by the non-availability of accurate and timely information regarding the location of damaged infrastructure and its extent. In this paper, we model the destruction in satellite imagery using a deep learning model employing a weakly-supervised approach. In stark contrast to previous approaches, instead of solving the problem as change detection (using pre and post-event images), we model to identify destruction itself using a single post-event image. To overcome the challenge of collecting pixel-level ground truth data mostly used during training, we only assume image-level labels, representing either destruction is present (at any location) in a given image or not. The proposed attention-based mechanism learns to identify the image-patches with destruction automatically under the sparsity constraint. Furthermore, to reduce false-positive and improve segmentation quality, a hard negative mining technique has been proposed that results in considerable improvement over baseline. To validate our approach, we have collected a new dataset containing destruction and non-destruction images from Indonesia, Yemen, Japan, and Pakistan. On testing-dataset, we obtained excellent destruction results with pixel-level accuracy of 93% and patch level accuracy of 91%. The source code and dataset will be made publicly available. .

[1]  Gui-Song Xia,et al.  AID: A Benchmark Data Set for Performance Evaluation of Aerial Scene Classification , 2016, IEEE Transactions on Geoscience and Remote Sensing.

[2]  Ryosuke Nakamura,et al.  Damage detection from aerial images via convolutional neural networks , 2017, 2017 Fifteenth IAPR International Conference on Machine Vision Applications (MVA).

[3]  Bohyung Han,et al.  Weakly Supervised Action Localization by Sparse Temporal Pooling Network , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[4]  Gang Wang,et al.  Progressive Attention Guided Recurrent Network for Salient Object Detection , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[5]  Christoph H. Lampert,et al.  Seed, Expand and Constrain: Three Principles for Weakly-Supervised Image Segmentation , 2016, ECCV.

[6]  Mubarak Shah,et al.  Real-World Anomaly Detection in Surveillance Videos , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[7]  Stuart P. D. Gill,et al.  A Comprehensive Analysis of Building Damage in the 12 January 2010 Mw7 Haiti Earthquake Using High-Resolution Satellite and Aerial Imagery , 2011 .

[8]  Qianqing Qin,et al.  Scene Classification Based on Multiscale Convolutional Neural Network , 2017, IEEE Transactions on Geoscience and Remote Sensing.

[9]  Fan Zhang,et al.  TreeUNet: Adaptive Tree convolutional neural networks for subdecimeter aerial image segmentation , 2019, ISPRS Journal of Photogrammetry and Remote Sensing.

[10]  Suha Kwak,et al.  Weakly Supervised Learning of Instance Segmentation With Inter-Pixel Relations , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Yang Shao,et al.  Detection of Urban Damage Using Remote Sensing and Machine Learning Algorithms: Revisiting the 2010 Haiti Earthquake , 2016, Remote. Sens..

[12]  Xiaoqiang Lu,et al.  Remote Sensing Image Scene Classification: Benchmark and State of the Art , 2017, Proceedings of the IEEE.

[13]  Zhuowen Tu,et al.  Deeply Supervised Salient Object Detection with Short Connections , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Lei Guo,et al.  Object Detection in Optical Remote Sensing Images Based on Weakly Supervised Learning and High-Level Feature Learning , 2015, IEEE Transactions on Geoscience and Remote Sensing.

[15]  Bernt Schiele,et al.  Simple Does It: Weakly Supervised Instance and Semantic Segmentation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Suha Kwak,et al.  Learning Pixel-Level Semantic Affinity with Image-Level Supervision for Weakly Supervised Semantic Segmentation , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[17]  Yoshimitsu Aoki,et al.  Disaster detection from aerial imagery with convolutional neural network , 2017, 2017 International Electronics Symposium on Knowledge Creation and Intelligent Computing (IES-KCIC).

[18]  Yu Zhang,et al.  What is and What is Not a Salient Object? Learning Salient Object Detector by Ensembling Linear Exemplar Regressors , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Matthieu Cord,et al.  WILDCAT: Weakly Supervised Learning of Deep ConvNets for Image Classification, Pointwise Localization and Segmentation , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Nicolas Brown,et al.  Mapping dead forest cover using a deep convolutional neural network and digital aerial photography , 2019, ISPRS Journal of Photogrammetry and Remote Sensing.

[21]  Raffay Hamid,et al.  Large-scale damage detection using satellite imagery , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Aydan Menderes,et al.  Automatic Detection of Damaged Buildings after Earthquake Hazard by Using Remote Sensing and Information Technologies , 2015 .

[23]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  A. Ng.  Feature selection, L1 vs. L2 regularization, and rotational invariance , 2004, Twenty-first international conference on Machine learning - ICML '04.

[25]  Shunichi Koshimura,et al.  Mapping of Building Damage of the 2011 Tohoku Earthquake Tsunami in Miyagi Prefecture , 2012 .

[26]  Gang Wang,et al.  Deep Level Sets for Salient Object Detection , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  Jie Shan,et al.  A comprehensive review of earthquake-induced building damage detection with remote sensing techniques , 2013 .

[28]  Han Dong,et al.  Building Damage Detection from Post-Event Aerial Imagery Using Single Shot Multibox Detector , 2019, Applied Sciences.

[29]  Xin Huang,et al.  Deep networks under scene-level supervision for multi-class geospatial object detection from remote sensing images , 2018, ISPRS Journal of Photogrammetry and Remote Sensing.