Computer Vision-Based Bridge Damage Detection Using Deep Convolutional Networks with Expectation Maximum Attention Module

Cracks and exposed steel bars are the main factors that affect the service life of bridges. It is necessary to detect the surface damage during regular bridge inspections. Due to the complex structure of bridges, automatically detecting bridge damage is a challenging task. In the field of crack classification and segmentation, convolutional neural networks have offer advantages, but ordinary networks cannot completely solve the environmental impact problems in reality. To further overcome these problems, in this paper a new algorithm to detect surface damage called EMA-DenseNet is proposed. The main contribution of this article is to redesign the structure of the densely connected convolutional networks (DenseNet) and add the expected maximum attention (EMA) module after the last pooling layer. The EMA module is obviously helpful to the bridge damage feature extraction. Besides, we use a new loss function which considers the connectivity of pixels, it has been proved to be effective in reducing the break point of fracture prediction and improving the accuracy. To train and test the model, we captured many images from multiple bridges located in Zhejiang (China), and then built a dataset of bridge damage images. First, experiments were carried out on an open concrete crack dataset. The mean pixel accuracy (MPA), mean intersection over union (MIoU), precision and frames per second (FPS) of the EMA-DenseNet are 87.42%, 92.59%, 81.97% and 25.4, respectively. Then we also conducted experiments on a more challenging bridge damage dataset, the MIoU, where MPA, precision and FPS were 79.87%, 86.35%, 74.70% and 14.6, respectively. Compared with the current state-of-the-art algorithms, the proposed algorithm is more accurate and robust in bridge damage detection.

[1]  Qinghua Han,et al.  Localization of acoustic emission sources in structural health monitoring of masonry bridge , 2015 .

[2]  Oral Büyüköztürk,et al.  Deep Learning‐Based Crack Damage Detection Using Convolutional Neural Networks , 2017, Comput. Aided Civ. Infrastructure Eng..

[3]  Guangpan Zhou,et al.  Structural Health Monitoring and Time-Dependent Effects Analysis of Self-Anchored Suspension Bridge with Extra-Wide Concrete Girder , 2018 .

[4]  Keith Worden,et al.  On switching response surface models, with applications to the structural health monitoring of bridges , 2018 .

[5]  Mohammad R. Jahanshahi,et al.  NB-CNN: Deep Learning-Based Crack Detection Using Convolutional Neural Network and Naïve Bayes Data Fusion , 2018, IEEE Transactions on Industrial Electronics.

[6]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Xiaochun Luo,et al.  Automatic Pixel‐Level Crack Detection and Measurement Using Fully Convolutional Network , 2018, Comput. Aided Civ. Infrastructure Eng..

[8]  Jian Sun,et al.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[9]  Robert J. Thomas,et al.  Comparison of deep convolutional neural networks and edge detectors for image-based crack detection in concrete , 2018, Construction and Building Materials.

[10]  Kristin J. Dana,et al.  Development of an autonomous bridge deck inspection robotic system , 2017, J. Field Robotics.

[11]  Wei Lu,et al.  Image-based concrete crack detection in tunnels using deep fully convolutional networks , 2020 .

[12]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[13]  George Papandreou,et al.  Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation , 2018, ECCV.

[14]  Hong Guan,et al.  Towards UAV-based bridge inspection systems: a review and an application perspective , 2015 .

[15]  Hui Li,et al.  Surface fatigue crack identification in steel box girder of bridges by a deep fusion convolutional neural network based on consumer-grade camera images , 2019 .

[16]  Lin Gao,et al.  Automated Pavement Crack Damage Detection Using Deep Multiscale Convolutional Features , 2020, Journal of Advanced Transportation.

[17]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[19]  Biao Ma,et al.  Automatic Tunnel Crack Detection Based on U-Net and a Convolutional Neural Network with Alternately Updated Clique , 2020, Sensors.

[20]  Xiaoxiao Li,et al.  Data mining algorithms for bridge health monitoring: Kohonen clustering and LSTM prediction approaches , 2020, The Journal of Supercomputing.

[21]  Young-Jin Cha,et al.  SDDNet: Real-Time Crack Segmentation , 2020, IEEE Transactions on Industrial Electronics.

[22]  Hui Li,et al.  Identification of spatio‐temporal distribution of vehicle loads on long‐span bridges using computer vision technology , 2016 .

[23]  Hong Liu,et al.  Expectation-Maximization Attention Networks for Semantic Segmentation , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[24]  Shuji Hashimoto,et al.  Fast crack detection method for large-size concrete surface images using percolation-based image processing , 2010, Machine Vision and Applications.

[25]  Hoon Sohn,et al.  An information modeling framework for bridge monitoring , 2017, Adv. Eng. Softw..

[26]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[27]  Roberto Cipolla,et al.  SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[28]  Mustafa Gul,et al.  Densely connected deep neural network considering connectivity of pixels for automatic crack detection , 2020 .

[29]  Calvin Coopmans,et al.  Fatigue Crack Detection Using Unmanned Aerial Systems in Under-Bridge Inspection , 2017 .

[30]  Yang Liu,et al.  Automated Pixel‐Level Pavement Crack Detection on 3D Asphalt Surfaces Using a Deep‐Learning Network , 2017, Comput. Aided Civ. Infrastructure Eng..

[31]  Abhinav Gupta,et al.  Non-local Neural Networks , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[32]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[33]  Mohd. Zaki Nuawi,et al.  Ultrasonic health monitoring in structural engineering: buildings and bridges , 2016 .

[34]  Yang Liu,et al.  Deep Learning-Based Fully Automated Pavement Crack Detection on 3D Asphalt Surfaces with an Improved CrackNet , 2018, J. Comput. Civ. Eng..

[35]  Emir Buza,et al.  Pavement crack detection using Otsu thresholding for image segmentation , 2018, 2018 41st International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO).

[36]  Liang Song,et al.  Faster region convolutional neural network for automated pavement distress detection , 2019, Road Materials and Pavement Design.

[37]  Jin-Won Nam,et al.  Development of Crack Detection System with Unmanned Aerial Vehicles and Digital Image Processing , 2015 .

[38]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[39]  Yanliang Gu,et al.  Automatic Crack Detection and Segmentation Using a Hybrid Algorithm for Road Distress Analysis , 2013, 2013 IEEE International Conference on Systems, Man, and Cybernetics.

[40]  Cao Vu Dung,et al.  Autonomous concrete crack detection using deep fully convolutional neural network , 2019, Automation in Construction.

[41]  Pi-Cheng Tung,et al.  The development of a mobile manipulator imaging system for bridge crack inspection , 2002 .

[42]  Young-Soo Park,et al.  An efficient image-based damage detection for cable surface in cable-stayed bridges , 2013 .

[43]  Weihua Sheng,et al.  Developing a crack inspection robot for bridge maintenance , 2011, 2011 IEEE International Conference on Robotics and Automation.

[44]  Hyoungkwan Kim,et al.  Encoder–decoder network for pixel‐level road crack detection in black‐box images , 2019, Comput. Aided Civ. Infrastructure Eng..

[45]  Cheng-Hsuan Yang,et al.  An Optimized Unmanned Aerial System for Bridge Inspection , 2015 .

[46]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[47]  Chung-Ming Yang,et al.  Thin crack observation in a reinforced concrete bridge pier test using image processing and analysis , 2015, Adv. Eng. Softw..

[48]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.