Detection of seismic façade damages with multi-temporal oblique aerial imagery

ABSTRACT Remote sensing images have long been recognized as useful for the detection of building damages, mainly due to their wide coverage, revisit capabilities and high spatial resolution. The majority of contributions aimed at identifying debris and rubble piles, as the main focus is to assess collapsed and partially collapsed structures. However, these approaches might not be optimal for the image classification of façade damages, where damages might appear in the form of spalling, cracks and collapse of small segments of the façade. A few studies focused their damage detection on the façades using only post-event images. Nonetheless, several studies achieved better performances in damage detection approaches when considering multi-temporal image data. Hence, in this work a multi-temporal façade damage detection is tested. The first objective is to optimally merge pre- and post-event aerial oblique imagery within a supervised classification approach using convolutional neural networks to detect façade damages. The second objective is related to the fact that façades are normally depicted in several views in aerial manned photogrammetric surveys; hence, different procedures combining these multi-view image data are also proposed and embedded in the image classification approach. Six multi-temporal approaches are compared against 3 mono-temporal ones. The results indicate the superiority of multi-temporal approaches (up to ~25% in f1-score) when compared to the mono-temporal ones. The best performing multi-temporal approach takes as input sextuples (3 views per epoch, per façade) within a late fusion approach to perform the image classification of façade damages. However, the detection of small damages, such as smaller cracks or smaller areas of spalling, remains challenging in this approach, mainly due to the low resolution (~0.14 m ground sampling distance) of the dataset used.

[1]  Paolo Gamba,et al.  Remote Sensing and Earthquake Damage Assessment: Experiences, Limits, and Perspectives , 2012, Proceedings of the IEEE.

[2]  R. Reulke,et al.  Remote Sensing and Spatial Information Sciences , 2005 .

[3]  Tao Liu,et al.  Deep convolutional neural network training enrichment using multi-view object-based analysis of Unmanned Aerial systems imagery for wetlands classification , 2018 .

[4]  Michael Kampffmeyer,et al.  Semantic Segmentation of Small Objects and Modeling of Uncertainty in Urban Remote Sensing Images Using Deep Convolutional Neural Networks , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[5]  M. Gerke,et al.  Automatic Structural Seismic Damage Assessment with Airborne Oblique Pictometry© Imagery , 2011 .

[6]  Amy Loutfi,et al.  Classification and Segmentation of Satellite Orthoimagery Using Convolutional Neural Networks , 2016, Remote. Sens..

[7]  Norman Kerle,et al.  Collaborative damage mapping for emergency response: the role of Cognitive Systems Engineering , 2013 .

[8]  Qingquan Li,et al.  A Novel 3D Building Damage Detection Method Using Multiple Overlapping UAV Images , 2014 .

[9]  James Hays,et al.  Localizing and Orienting Street Views Using Overhead Imagery , 2016, ECCV.

[10]  Gui-Song Xia,et al.  Accurate Annotation of Remote Sensing Images via Active Spectral Clustering with Little Expert Knowledge , 2015, Remote. Sens..

[11]  United Nations.  OFFICE FOR THE COORDINATION OF HUMANITARIAN AFFAIRS , 2000 .

[12]  Richard Lepage,et al.  Fast and Efficient Evaluation of Building Damage From Very High Resolution Optical Satellite Images , 2014, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[13]  Keisuke Nemoto,et al.  Effective Use of Dilated Convolutions for Segmenting Small Object Instances in Remote Sensing Imagery , 2017, 2018 IEEE Winter Conference on Applications of Computer Vision (WACV).

[14]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[15]  Xinwu Li,et al.  A New Approach to Collapsed Building Extraction Using RADARSAT-2 Polarimetric SAR Imagery , 2012, IEEE Geoscience and Remote Sensing Letters.

[16]  P. Axelsson.  DEM Generation from Laser Scanner Data Using Adaptive TIN Models , 2000 .

[17]  Francesco Carlo Nex,et al.  Usability of aerial video footage for 3-D scene reconstruction and structural damage assessment , 2018, Natural Hazards and Earth System Sciences.

[18]  Alexandre Boulch,et al.  Urban Change Detection for Multispectral Earth Observation Using Convolutional Neural Networks , 2018, IGARSS 2018 - 2018 IEEE International Geoscience and Remote Sensing Symposium.

[19]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Yifei Zhang,et al.  Automated Landslides Detection for Mountain Cities Using Multi-Temporal Remote Sensing Imagery , 2018, Sensors.

[21]  D. Duarte,et al.  Debris, rubble piles and façade damage detection using multi-resolution optical remote sensing imagery , 2020 .

[22]  Karsten Schulz,et al.  Building damage assessment in decimeter resolution SAR imagery: A future perspective , 2011, 2011 Joint Urban Remote Sensing Event.

[23]  Thomas A. Funkhouser,et al.  Dilated Residual Networks , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Gui-Song Xia,et al.  Transferring Deep Convolutional Neural Networks for the Scene Classification of High-Resolution Remote Sensing Imagery , 2015, Remote. Sens..

[25]  Dongmei Chen,et al.  Change detection from remotely sensed images: From pixel-based to object-based approaches , 2013 .

[26]  Herbert Freeman,et al.  Determining the minimum-area encasing rectangle for an arbitrary closed curve , 1975, CACM.

[27]  Friedrich Fraundorfer,et al.  Automatic Annotation of Airborne Images by Label Propagation Based on a Bayesian-CRF Model , 2019, Remote. Sens..

[28]  Sepp Hochreiter,et al.  Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) , 2015, ICLR.

[29]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  George Vosselman,et al.  SATELLITE IMAGE CLASSIFICATION OF BUILDING DAMAGES USING AIRBORNE AND SATELLITE IMAGE SAMPLES IN A DEEP LEARNING APPROACH , 2018 .

[31]  Alfred Stein,et al.  Deep Fully Convolutional Networks for the Detection of Informal Settlements in VHR Images , 2017, IEEE Geoscience and Remote Sensing Letters.

[32]  Chi Zhang,et al.  Detecting Large-Scale Urban Land Cover Changes from Very High Resolution Remote Sensing Images Using CNN-Based Classification , 2019, ISPRS Int. J. Geo Inf..

[33]  Pierre Grussenmeyer,et al.  Oblique Aerial Photography Tool for Building Inspection and Damage Assessment , 2014 .

[34]  George Vosselman,et al.  Automated planimetric quality control in high accuracy airborne laser scanning surveys , 2012 .

[35]  Duk-jin Kim,et al.  Damage-Mapping Algorithm Based on Coherence Model Using Multitemporal Polarimetric–Interferometric SAR Data , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[36]  C. Atzberger,et al.  Evaluation of semi-empirical BRDF models inverted against multi-angle data from a digital airborne frame camera for enhancing forest type classification , 2014 .

[37]  Michele Volpi,et al.  Dense Semantic Labeling of Subdecimeter Resolution Images With Convolutional Neural Networks , 2016, IEEE Transactions on Geoscience and Remote Sensing.

[38]  George Vosselman,et al.  Multi-Resolution Feature Fusion for Image Classification of Building Damages with Convolutional Neural Networks , 2018, Remote. Sens..

[39]  Hou Jiang,et al.  Multi-Scale Residual Convolutional Neural Network for Haze Removal of Remote Sensing Images , 2018, Remote. Sens..

[40]  Ashbindu Singh,et al.  Review Article Digital change detection techniques using remotely-sensed data , 1989 .

[41]  Sudan Xu,et al.  Segment-Based Classification of Damaged Building Roofs in Aerial Laser Scanning Data , 2013, IEEE Geoscience and Remote Sensing Letters.

[42]  A. Vetrivel,et al.  Disaster damage detection through synergistic use of deep learning and 3D point cloud features derived from very high resolution oblique aerial images, and multiple-kernel-learning , 2017, ISPRS Journal of Photogrammetry and Remote Sensing.

[43]  Jefersson Alex dos Santos,et al.  Towards better exploiting convolutional neural networks for remote sensing scene classification , 2016, Pattern Recognit..

[44]  A. Vetrivel,et al.  POTENTIAL OF MULTI-TEMPORAL OBLIQUE AIRBORNE IMAGERY FOR STRUCTURAL DAMAGE ASSESSMENT , 2016 .

[45]  George Vosselman,et al.  Damage detection on building façades using multi-temporal aerial oblique imagery , 2019 .

[46]  Jie Shan,et al.  A comprehensive review of earthquake-induced building damage detection with remote sensing techniques , 2013 .

[47]  Bertrand Le Saux,et al.  Beyond RGB: Very High Resolution Urban Remote Sensing With Multimodal Deep Networks , 2017, ISPRS Journal of Photogrammetry and Remote Sensing.

[48]  Bertrand Le Saux,et al.  Semantic Segmentation of Earth Observation Data Using Multimodal and Multi-scale Deep Networks , 2016, ACCV.

[49]  Vladlen Koltun,et al.  Multi-Scale Context Aggregation by Dilated Convolutions , 2015, ICLR.

[50]  anonymous,et al.  Comprehensive review , 2019 .

[51]  Diego González-Aguilera,et al.  Terrestrial laser scanning intensity data applied to damage detection for historical buildings , 2010 .

[52]  George Vosselman,et al.  Towards a more efficient detection of earthquake induced façade damages using oblique UAV imagery. , 2017 .

[53]  XuXin,et al.  Multi-view learning overview , 2017 .

[54]  D. Lu,et al.  Change detection techniques , 2004 .

[55]  Qing Wang,et al.  Change detection based on Faster R-CNN for high-resolution remote sensing images , 2018, Remote Sensing Letters.

[56]  H. Sui,et al.  Detecting building façade damage from oblique aerial images using local symmetry feature and the Gini Index , 2017 .

[57]  Pierre Alliez,et al.  Convolutional Neural Networks for Large-Scale Remote-Sensing Image Classification , 2017, IEEE Transactions on Geoscience and Remote Sensing.

[58]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[59]  Diogo Duarte,et al.  Towards Real-Time Building Damage Mapping with Low-Cost UAV Solutions , 2019, Remote. Sens..

[60]  Fan Zhang,et al.  Deep Convolutional Neural Networks for Hyperspectral Image Classification , 2015, J. Sensors.

[61]  Nicholas J. Tate,et al.  A critical synthesis of remotely sensed optical image change detection techniques , 2015 .

[62]  Norman Kerle,et al.  UAV-based urban structural damage assessment using object-based image analysis and semantic reasoning , 2014 .

[63]  Tao Liu,et al.  A fully learnable context-driven object-based model for mapping land cover using multi-view data from unmanned aircraft systems , 2018, Remote Sensing of Environment.

[64]  William F. Fagan,et al.  Capturing Damage Assessment with a Spatial Video: An Example of a Building and Street-Scale Analysis of Tornado-Related Mortality in Joplin, Missouri, 2011 , 2013 .

[65]  Shiliang Sun,et al.  Multi-view learning overview: Recent progress and new challenges , 2017, Inf. Fusion.

[66]  Tao Liu,et al.  Multi-view object-based classification of wetland land covers using unmanned aircraft system images , 2018, Remote Sensing of Environment.