Deep Transfer Learning for Image‐Based Structural Damage Recognition

This article implements the state‐of‐the‐art deep learning technologies for a civil engineering application, namely recognition of structural damage from images. Inspired by ImageNet Challenge and the development of computer hardware, the concept of Structural ImageNet is proposed herein with four naïve baseline recognition tasks: component type identification, spalling condition check, damage level evaluation, and damage type determination. A relatively small number of images (2,000) are selected from the Structural ImageNet and manually labeled according to the four recognition tasks. In order to avoid overfitting, Transfer Learning (TL) based on VGGNet (Visual Geometry Group) is introduced and applied using two different strategies, namely feature extractor and fine‐tuning. Two experiments are designed based on properties of these two strategies to find the relative optimal model parameters and scope of application. Models obtained by both strategies indicate the promising recognition results and different application potentials where feature extractor and fine‐tuning can be respectively used for preliminary analysis and for further improvement. These results also reveal the potential uses of deep TL in image‐based structural damage recognition.

[1]  Maria Q. Feng,et al.  Experimental validation of cost-effective vision-based structural health monitoring , 2017 .

[2]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[3]  Sangjoon Park,et al.  Applications of laser scanning to structures in laboratory tests and field surveys , 2014 .

[4]  Oral Büyüköztürk,et al.  Autonomous Structural Visual Inspection Using Region‐Based Deep Learning for Detecting Multiple Damage Types , 2018, Comput. Aided Civ. Infrastructure Eng..

[5]  A. Vetrivel,et al.  Disaster damage detection through synergistic use of deep learning and 3D point cloud features derived from very high resolution oblique aerial images, and multiple-kernel-learning , 2017, ISPRS Journal of Photogrammetry and Remote Sensing.

[6]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[7]  Guido Morgenthal,et al.  Quality Assessment of Unmanned Aerial Vehicle (UAV) Based Visual Inspection of Structures , 2014 .

[8]  Mani Golparvar-Fard,et al.  Target‐free approach for vision‐based structural system identification using consumer‐grade cameras , 2016 .

[9]  Rob Fergus,et al.  Visualizing and Understanding Convolutional Networks , 2013, ECCV.

[10]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[11]  Andrew S. Whittaker,et al.  Performance of reinforced concrete buildings during the August 17, 1999 Kocaeli, Turkey earthquake, and seismic design and construction practise in Turkey , 2003 .

[12]  Ramprasaath R. Selvaraju,et al.  Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[13]  Oral Büyüköztürk,et al.  Deep Learning‐Based Crack Damage Detection Using Convolutional Neural Networks , 2017, Comput. Aided Civ. Infrastructure Eng..

[14]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Bolei Zhou,et al.  Learning Deep Features for Discriminative Localization , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[17]  Chunsun Zhang,et al.  An Unmanned Aerial Vehicle‐Based Imaging System for 3D Measurement of Unpaved Road Surface Distresses 1 , 2012, Comput. Aided Civ. Infrastructure Eng..

[18]  Yoshua Bengio,et al.  Deep Learning of Representations for Unsupervised and Transfer Learning , 2011, ICML Unsupervised and Transfer Learning.

[19]  Carlos Guestrin,et al.  "Why Should I Trust You?": Explaining the Predictions of Any Classifier , 2016, ArXiv.

[20]  Chul Min Yeum,et al.  Vision‐Based Automated Crack Detection for Bridge Inspection , 2015, Comput. Aided Civ. Infrastructure Eng..

[21]  Hojjat Adeli,et al.  Parallel Structural Analysis Using Threads , 2008 .

[22]  Mani Golparvar-Fard,et al.  Image-Based Automated 3D Crack Detection for Post-disaster Building Assessment , 2014, J. Comput. Civ. Eng..

[23]  Luca Maria Gambardella,et al.  Max-pooling convolutional neural networks for vision-based hand gesture recognition , 2011, 2011 IEEE International Conference on Signal and Image Processing Applications (ICSIPA).

[24]  L. Girardi,et al.  Seismic performance , 2018, Astronomy & Astrophysics.

[25]  Mohamed A. El-Sayed,et al.  Automated Edge Detection Using Convolutional Neural Network , 2013 .

[26]  Yoshua Bengio,et al.  Object Recognition with Gradient-Based Learning , 1999, Shape, Contour and Grouping in Computer Vision.

[27]  Qiang Yang,et al.  A Survey on Transfer Learning , 2010, IEEE Transactions on Knowledge and Data Engineering.

[28]  Ronald M. Summers,et al.  Deep Convolutional Neural Networks for Computer-Aided Detection: CNN Architectures, Dataset Characteristics and Transfer Learning , 2016, IEEE Transactions on Medical Imaging.

[29]  Geoffrey E. Hinton,et al.  Rectified Linear Units Improve Restricted Boltzmann Machines , 2010, ICML.

[30]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[31]  Lawrence D. Jackel,et al.  Backpropagation Applied to Handwritten Zip Code Recognition , 1989, Neural Computation.

[32]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[33]  Ling Shao,et al.  Transfer Learning for Visual Categorization: A Survey , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[34]  Reinhold Huber-Mörk,et al.  Convolutional Neural Networks for Steel Surface Defect Detection from Photometric Stereo Images , 2014, ISVC.

[35]  Li-Rong Dai,et al.  Deep Bottleneck Feature for Image Classification , 2015, ICMR.

[36]  Paul W. Fieguth,et al.  A review on computer vision based defect detection and condition assessment of concrete and asphalt civil infrastructure , 2015, Adv. Eng. Informatics.

[37]  Abhishek Das,et al.  Grad-CAM: Why did you say that? , 2016, ArXiv.

[38]  Ivan Laptev,et al.  Learning and Transferring Mid-level Image Representations Using Convolutional Neural Networks , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[39]  Jack P. Moehle,et al.  Seismic Design of Reinforced Concrete Buildings , 2014 .

[40]  Khalid M. Mosalam,et al.  Seismic Performance of Reinforced-Concrete Stairways during the 2008 Wenchuan Earthquake , 2013 .

[41]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[42]  Yang Liu,et al.  Automated Pixel‐Level Pavement Crack Detection on 3D Asphalt Surfaces Using a Deep‐Learning Network , 2017, Comput. Aided Civ. Infrastructure Eng..

[43]  Abhishek Das,et al.  Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[44]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[45]  Moncef Gabbouj,et al.  Real-time vibration-based structural damage detection using one-dimensional convolutional neural networks , 2017 .