Machine vision-based surface crack analysis for transportation infrastructure

Abstract Cracks undermine the structural health of transportation infrastructure. Machine vision-based surface crack analysis is to process infrastructure inspection data collected by imaging devices for identifying the presence, location, and extent of cracks, classifying the corresponding severity levels, and eventually predicting their growth. Unlike the fragmented qualitative discussions on machine vision-based crack analysis methods in existing studies, this paper reviews the state of the art and practice of various machine vision solutions under different operating conditions in a fine-grained quantitative way, systematically describing the strengths and limitations of deep learning over other solutions. Moreover, the applicability assessment is implemented to describe the deployment and optimization of deep learning in five crack analysis tasks: image classification, object detection, pixel segmentation, geometric scale quantification, and growth prediction. At last, the challenges faced and corresponding breakthrough directions are summarized, respectively, driving further development of deep learning to assist more sophisticated maintenance decisions.

[1]  Fan Meng,et al.  Automatic Road Crack Detection Using Random Structured Forests , 2016, IEEE Transactions on Intelligent Transportation Systems.

[2]  Xiangtao Zheng,et al.  Sample and Structure-Guided Network for Road Crack Detection , 2019, IEEE Access.

[3]  Kechen Song,et al.  Deep Metric Learning-Based for Multi-Target Few-Shot Pavement Distress Classification , 2021, IEEE Transactions on Industrial Informatics.

[4]  Yi Wang,et al.  Automatic Bridge Crack Detection Using a Convolutional Neural Network , 2019, Applied Sciences.

[5]  Mohammad Mahmudur Rahman,et al.  Use of a Self-Organizing Map for Crack Detection in Highly Textured Pavement Images , 2015 .

[6]  Hyoungkwan Kim,et al.  Patch-Based Crack Detection in Black Box Images Using Convolutional Neural Networks , 2019, J. Comput. Civ. Eng..

[7]  Kelvin C. P. Wang,et al.  Pavement Crack Width Measurement Based on Laplace's Equation for Continuity and Unambiguity , 2018, Comput. Aided Civ. Infrastructure Eng..

[8]  Minghai Yao,et al.  Improving the Efficiency of Encoder-Decoder Architecture for Pixel-Level Crack Detection , 2019, IEEE Access.

[9]  Zheng Tong,et al.  Pavement defect detection with fully convolutional network and an uncertainty framework , 2020, Comput. Aided Civ. Infrastructure Eng..

[10]  Zhun Fan,et al.  Automatic Pavement Crack Detection Based on Structured Prediction with the Convolutional Neural Network , 2018, ArXiv.

[11]  Qingquan Li,et al.  CrackTree: Automatic crack detection from pavement images , 2012, Pattern Recognit. Lett..

[12]  Yu Liu,et al.  Automatic Pavement Crack Detection by Multi-Scale Image Fusion , 2019, IEEE Transactions on Intelligent Transportation Systems.

[13]  Yu-Hsing Wang,et al.  Anomaly detection of defects on concrete structures with the convolutional autoencoder , 2020, Adv. Eng. Informatics.

[14]  Wei Ren,et al.  Automatic recognition and analysis system of asphalt pavement cracks using interleaved low-rank group convolution hybrid deep network and SegNet fusing dense condition random field , 2021 .

[15]  Won-Hwa Hong,et al.  Application of Crack Identification Techniques for an Aging Concrete Bridge Inspection Using an Unmanned Aerial Vehicle , 2018, Sensors.

[16]  Shirley J. Dyke,et al.  Automated region-of-interest localization and classification for vision-based visual assessment of civil infrastructure , 2019 .

[17]  Chenkai Shen,et al.  A real-time detection approach for bridge cracks based on YOLOv4-FPM , 2021 .

[18]  Li Li,et al.  DeepCrack: A deep hierarchical feature learning architecture for crack segmentation , 2019, Neurocomputing.

[19]  Antonio Torralba,et al.  LabelMe: A Database and Web-Based Tool for Image Annotation , 2008, International Journal of Computer Vision.

[20]  Yong Hu,et al.  Automatic Pavement Crack Detection Using Texture and Shape Descriptors , 2010 .

[21]  Gholamreza Ghodrati Amiri,et al.  Unsupervised Structural Damage Detection Technique Based on a Deep Convolutional Autoencoder , 2021 .

[22]  Lin Gao,et al.  Automated Pavement Crack Damage Detection Using Deep Multiscale Convolutional Features , 2020, Journal of Advanced Transportation.

[23]  Yuchuan Du,et al.  Pavement distress detection and classification based on YOLO network , 2020, International Journal of Pavement Engineering.

[24]  Wei Li,et al.  Detection of sealed and unsealed cracks with complex backgrounds using deep convolutional neural network , 2019, Automation in Construction.

[25]  Nikolaos Doulamis,et al.  Combined Convolutional Neural Networks and Fuzzy Spectral Clustering for Real Time Crack Detection in Tunnels , 2018, 2018 25th IEEE International Conference on Image Processing (ICIP).

[26]  Donghan Lee,et al.  Robust Concrete Crack Detection Using Deep Learning-Based Semantic Segmentation , 2019, International Journal of Aeronautical and Space Sciences.

[27]  Yang Liu,et al.  Deep Learning-Based Fully Automated Pavement Crack Detection on 3D Asphalt Surfaces with an Improved CrackNet , 2018, J. Comput. Civ. Eng..

[28]  Xinxin Niu,et al.  Automatic Recognition of Pavement Surface Crack Based on BP Neural Network , 2008, 2008 International Conference on Computer and Electrical Engineering.

[29]  Sumathi Poobal,et al.  Crack detection using image processing: A critical review and analysis , 2017, Alexandria Engineering Journal.

[30]  Kaige Zhang,et al.  CrackGAN: Pavement Crack Detection Using Partially Accurate Ground Truths Based on Generative Adversarial Learning , 2020, IEEE Transactions on Intelligent Transportation Systems.

[31]  Nima Safaei,et al.  An Automatic Image Processing Algorithm Based on Crack Pixel Density for Pavement Crack Detection and Classification , 2021, International Journal of Pavement Research and Technology.

[32]  Hongwei Huang,et al.  Deep learning–based image instance segmentation for moisture marks of shield tunnel lining , 2020 .

[33]  AmhazRabih,et al.  Automatic Crack Detection on Two-Dimensional Pavement Images , 2016 .

[34]  Nikolaos Doulamis,et al.  Deep Convolutional Neural Networks for efficient vision based tunnel inspection , 2015, 2015 IEEE International Conference on Intelligent Computer Communication and Processing (ICCP).

[35]  Ignacio Parra,et al.  Adaptive Road Crack Detection System by Pavement Classification , 2011, Sensors.

[36]  Dinesh Rajan,et al.  Concrete crack detection using context‐aware deep semantic segmentation network , 2019, Comput. Aided Civ. Infrastructure Eng..

[37]  Soojin Cho,et al.  Automated Vision-Based Detection of Cracks on Concrete Surfaces Using a Deep Learning Technique , 2018, Sensors.

[38]  Yoshihide Sekimoto,et al.  Road Damage Detection and Classification Using Deep Neural Networks with Smartphone Images , 2018, Comput. Aided Civ. Infrastructure Eng..

[39]  Dongming Zhang,et al.  Deep learning based image recognition for crack and leakage defects of metro shield tunnel , 2018, Tunnelling and Underground Space Technology.

[40]  H. Abdul Razak,et al.  Fault diagnosis on beam-like structures from modal parameters using artificial neural networks , 2015 .

[41]  Bin Song,et al.  Research on abnormal detection of one-class support vector machine based on ensemble cooperative semi-supervised learning , 2019, Journal of Physics: Conference Series.

[42]  Linbing Wang,et al.  The State-of-the-Art Review on Applications of Intrusive Sensing, Image Processing Techniques, and Machine Learning Methods in Pavement Monitoring and Analysis , 2020 .

[43]  Qiangwei Liu,et al.  Automatic crack recognition for concrete bridges using a fully convolutional neural network and naive Bayes data fusion based on a visual detection system , 2020, Measurement Science and Technology.

[44]  Ling Liu,et al.  Crack Detection of Concrete Pavement With Cross-Entropy Loss Function and Improved VGG16 Network Model , 2020, IEEE Access.

[45]  Wenming Cao,et al.  Review of Pavement Defect Detection Methods , 2020, IEEE Access.

[46]  Hyoungkwan Kim,et al.  Encoder–decoder network for pixel‐level road crack detection in black‐box images , 2019, Comput. Aided Civ. Infrastructure Eng..

[47]  Alexei Sourin,et al.  Unsupervised Surface Defect Detection Using Deep Autoencoders and Data Augmentation , 2018, 2018 International Conference on Cyberworlds (CW).

[48]  Lin Zhang,et al.  A One-Shot Texture-Perceiving Generative Adversarial Network for Unsupervised Surface Inspection , 2021, ICIP 2021.

[49]  Oral Büyüköztürk,et al.  Deep Learning‐Based Crack Damage Detection Using Convolutional Neural Networks , 2017, Comput. Aided Civ. Infrastructure Eng..

[50]  Biao Ma,et al.  Automatic Tunnel Crack Detection Based on U-Net and a Convolutional Neural Network with Alternately Updated Clique , 2020, Sensors.

[51]  Jialing Zou,et al.  Rock Image Segmentation of Improved Semi-supervised SVM–FCM Algorithm Based on Chaos , 2020, Circuits Syst. Signal Process..

[52]  Dimos Polyzois,et al.  Deep learning-based automatic volumetric damage quantification using depth camera , 2019, Automation in Construction.

[53]  Mohammad R. Jahanshahi,et al.  NB-CNN: Deep Learning-Based Crack Detection Using Convolutional Neural Network and Naïve Bayes Data Fusion , 2018, IEEE Transactions on Industrial Electronics.

[54]  Jie Gao,et al.  Recognition of asphalt pavement crack length using deep convolutional neural networks , 2018 .

[55]  Paul W. Fieguth,et al.  A review on computer vision based defect detection and condition assessment of concrete and asphalt civil infrastructure , 2015, Adv. Eng. Informatics.

[56]  Ikhlas Abdel-Qader,et al.  ANALYSIS OF EDGE-DETECTION TECHNIQUES FOR CRACK IDENTIFICATION IN BRIDGES , 2003 .

[57]  Natarajan Meghanathan,et al.  Seed-Based Approach for Automated Crack Detection from Pavement Images , 2016 .

[58]  Tianrui Li,et al.  Generative Adversarial Networks for Road Crack Image Segmentation , 2019, 2019 International Joint Conference on Neural Networks (IJCNN).

[59]  Xuefeng Zhao,et al.  Automatic pixel‐level multiple damage detection of concrete structure using fully convolutional network , 2019, Comput. Aided Civ. Infrastructure Eng..

[60]  Jian Zhang,et al.  Zernike‐moment measurement of thin‐crack width in images enabled by dual‐scale deep learning , 2018, Comput. Aided Civ. Infrastructure Eng..

[61]  Billie F. Spencer,et al.  Advances in Computer Vision-Based Civil Infrastructure Inspection and Monitoring , 2019, Engineering.

[62]  Zhihong Man,et al.  Dynamic neural modeling of fatigue crack growth process in ductile alloys , 2016, Inf. Sci..

[63]  Yufeng Zhang,et al.  Attribute-based structural damage identification by few-shot meta learning with inter-class knowledge transfer , 2020, Structural Health Monitoring.

[64]  Kelvin C. P. Wang,et al.  Pixel-Level Cracking Detection on 3D Asphalt Pavement Images Through Deep-Learning- Based CrackNet-V , 2020, IEEE Transactions on Intelligent Transportation Systems.

[65]  Devin K. Harris,et al.  Increasing the robustness of material-specific deep learning models for crack detection across different materials , 2020 .

[66]  Limao Zhang,et al.  A spatial-channel hierarchical deep learning network for pixel-level automated crack detection , 2020 .

[67]  Yuchun Huang,et al.  Pavement crack detection using hessian structure propagation , 2021, Adv. Eng. Informatics.

[68]  Ye Zhao,et al.  Graph-Based Semi-supervised Learning for Fault Detection and Classification in Solar Photovoltaic Arrays , 2015, IEEE Transactions on Power Electronics.

[69]  Tania Stathaki,et al.  Automatic crack detection for tunnel inspection using deep learning and heuristic image post-processing , 2018, Applied Intelligence.

[70]  Qiang Li,et al.  Matched Filtering Algorithm for Pavement Cracking Detection , 2013 .

[71]  Sherif Yehia,et al.  PCA-Based algorithm for unsupervised bridge crack detection , 2006, Adv. Eng. Softw..

[72]  Young-Jin Cha,et al.  Hybrid pixel-level concrete crack segmentation and quantification across complex backgrounds using deep learning , 2020 .

[73]  Devin K. Harris,et al.  A big data analytics strategy for scalable urban infrastructure condition assessment using semi-supervised multi-transform self-training , 2020 .

[74]  TG Ritto,et al.  Digital twin, physics-based model, and machine learning applied to damage detection in structures , 2020, Mechanical Systems and Signal Processing.

[75]  Carlos Balaguer,et al.  Intelligent robotic system for autonomous crack detection and caracterization in concrete tunnels , 2017, 2017 IEEE International Conference on Autonomous Robot Systems and Competitions (ICARSC).

[76]  Kelvin C. P. Wang,et al.  Cracking Classification Using Minimum Rectangular Cover-Based Support Vector Machine , 2017, J. Comput. Civ. Eng..

[77]  Yicheng Li,et al.  A Fast Detection Method via Region‐Based Fully Convolutional Neural Networks for Shield Tunnel Lining Defects , 2018, Comput. Aided Civ. Infrastructure Eng..

[78]  Chaobo Zhang,et al.  Concrete bridge surface damage detection using a single‐stage detector , 2019, Comput. Aided Civ. Infrastructure Eng..

[79]  Abbas Taheri,et al.  Pre-Peak and Post-Peak Rock Strain Characteristics During Uniaxial Compression by 3D Digital Image Correlation , 2016, Rock Mechanics and Rock Engineering.

[80]  Ju Huyan,et al.  Illumination Compensation Model with k -Means Algorithm for Detection of Pavement Surface Cracks with Shadow , 2020, J. Comput. Civ. Eng..

[81]  Xiaojin Zhu,et al.  Introduction to Semi-Supervised Learning , 2009, Synthesis Lectures on Artificial Intelligence and Machine Learning.

[82]  Wenzong Wang,et al.  Segment-based pavement crack quantification , 2019, Automation in Construction.

[83]  Yang Liu,et al.  Automated Pixel‐Level Pavement Crack Detection on 3D Asphalt Surfaces Using a Deep‐Learning Network , 2017, Comput. Aided Civ. Infrastructure Eng..

[84]  Mustafa Gul,et al.  A cost effective solution for pavement crack inspection using cameras and deep neural networks , 2020 .

[85]  Xiaochun Luo,et al.  Automatic Pixel‐Level Crack Detection and Measurement Using Fully Convolutional Network , 2018, Comput. Aided Civ. Infrastructure Eng..

[86]  Yoshihide Sekimoto,et al.  Road Damage Detection and Classification Using Deep Neural Networks with Smartphone Images , 2018, Comput. Aided Civ. Infrastructure Eng..

[87]  Liene Some.  Automatic image-based road crack detection methods , 2016 .

[88]  Wei Lu,et al.  Image-based concrete crack detection in tunnels using deep fully convolutional networks , 2020 .

[89]  Tim Broyd,et al.  Digital twin and its implementations in the civil engineering sector , 2021 .

[90]  Yanan Dong,et al.  A Deep-Learning-Based Multiple Defect Detection Method for Tunnel Lining Damages , 2019, IEEE Access.

[91]  Timon Rabczuk,et al.  Transfer learning enhanced physics informed neural network for phase-field modeling of fracture , 2019, Theoretical and Applied Fracture Mechanics.

[92]  Paulo Lobato Correia,et al.  CrackIT — An image processing toolbox for crack detection and characterization , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[93]  Mohan Rajesh Elara,et al.  Self-reconfigurable façade-cleaning robot equipped with deep-learning-based crack detection based on convolutional neural networks , 2019 .

[94]  Billie F. Spencer,et al.  Deep Learning-based Damage Detection of Miter Gates Using Synthetic Imagery from Computer Graphics , 2019 .

[95]  Li Feng,et al.  A bridge crack image detection and classification method based On climbing robot , 2016, 2016 35th Chinese Control Conference (CCC).

[96]  Heng-Da Cheng,et al.  Self-Supervised Structure Learning for Crack Detection Based on Cycle-Consistent Generative Adversarial Networks , 2020, J. Comput. Civ. Eng..

[97]  Gye-Chun Cho,et al.  Lightweight pixel-wise segmentation for efficient concrete crack detection using hierarchical convolutional neural network , 2021, Smart Materials and Structures.

[98]  Liang Song,et al.  Faster region convolutional neural network for automated pavement distress detection , 2019, Road Materials and Pavement Design.

[99]  Takayuki Okatani,et al.  A vision-based method for crack detection in gusset plate welded joints of steel bridges using deep convolutional neural networks , 2019, Automation in Construction.

[100]  Yan Wang,et al.  A new rail crack detection method using LSTM network for actual application based on AE technology , 2018, Applied Acoustics.

[101]  Young-Jin Cha,et al.  Subsurface damage detection of a steel bridge using deep learning and uncooled micro-bolometer , 2019, Construction and Building Materials.

[102]  M. Leo,et al.  Analysis of the Displacement Field of Soft Rock Samples During UCS Tests by Means of a Computer Vision Technique , 2019, Rock Mechanics and Rock Engineering.

[103]  Siddhartha Kumar Khaitan,et al.  Deep Convolutional Neural Networks with transfer learning for computer vision-based data-driven pavement distress detection , 2017 .

[104]  Jong-Myon Kim,et al.  Crack Classification of a Pressure Vessel Using Feature Selection and Deep Learning Methods , 2018, Sensors.

[105]  Yang Liu,et al.  Automated Pixel‐Level Pavement Crack Detection on 3D Asphalt Surfaces with a Recurrent Neural Network , 2018, Comput. Aided Civ. Infrastructure Eng..

[106]  Wei Song,et al.  Deep learning-based roadway crack classification using laser-scanned range images: A comparative study on hyperparameter selection , 2020, Automation in Construction.

[107]  Seunghyun Choi,et al.  Development of the Road Pavement Deterioration Model Based on the Deep Learning Method , 2019 .

[108]  Yuanjie Zheng,et al.  Prediction Method of Three‐Dimensional Crack Propagation Path Based on Deep Learning Application , 2020, Advanced Engineering Materials.

[109]  Robert J. Thomas,et al.  SDNET2018: An annotated image dataset for non-contact concrete crack detection using deep convolutional neural networks , 2018, Data in brief.

[110]  Branko Glisic,et al.  Crack detection and characterization techniques—An overview , 2014 .

[111]  Robert J. Thomas,et al.  Comparison of deep convolutional neural networks and edge detectors for image-based crack detection in concrete , 2018, Construction and Building Materials.

[112]  Naohiko Kohtake,et al.  Non-parametric Prediction Interval Estimate for Uncertainty Quantification of the Prediction of Road Pavement Deterioration , 2018, 2018 21st International Conference on Intelligent Transportation Systems (ITSC).

[113]  Ning Wang,et al.  New Crack Detection Method for Bridge Inspection Using UAV Incorporating Image Processing , 2018, Journal of Aerospace Engineering.

[114]  Kelvin C. P. Wang,et al.  Methodology for Accurate AASHTO PP67-10–Based Cracking Quantification Using 1-mm 3D Pavement Images , 2017 .

[115]  Dimitrios Giagopoulos,et al.  Structural Health Monitoring using deep learning with optimal finite element model generated data , 2020 .

[116]  Ying Chen,et al.  Ensemble of Deep Convolutional Neural Networks for Automatic Pavement Crack Detection and Measurement , 2020, Coatings.

[117]  Yun-Kyu An,et al.  Deep super resolution crack network (SrcNet) for improving computer vision–based automated crack detectability in in situ bridges , 2020, Structural Health Monitoring.

[118]  Xiaowei Luo,et al.  An integrated approach to automatic pixel-level crack detection and quantification of asphalt pavement , 2020, Automation in Construction.

[119]  Erol Tutumluer,et al.  Automated crack severity level detection and classification for ballastless track slab using deep convolutional neural network , 2021 .

[120]  Sung-Han Sim,et al.  A Novelty Detection Approach for Tendons of Prestressed Concrete Bridges Based on a Convolutional Autoencoder and Acceleration Data , 2019, Sensors.

[121]  Fan Yang,et al.  Feature Pyramid and Hierarchical Boosting Network for Pavement Crack Detection , 2019, IEEE Transactions on Intelligent Transportation Systems.

[122]  Mahindra Rautela,et al.  Ultrasonic guided wave based structural damage detection and localization using model assisted convolutional and recurrent neural networks , 2020, Expert Syst. Appl..

[123]  Jure Skvarč,et al.  Segmentation-based deep-learning approach for surface-defect detection , 2019, Journal of Intelligent Manufacturing.

[124]  Aditya M. Deshpande,et al.  One-Shot Recognition of Manufacturing Defects in Steel Surfaces , 2020, Procedia Manufacturing.

[125]  Ye Lu,et al.  Concrete crack detection with handwriting script interferences using faster region‐based convolutional neural network , 2019, Comput. Aided Civ. Infrastructure Eng..

[126]  Heng Zhang,et al.  A UAV-based crack inspection system for concrete bridge monitoring , 2017, 2017 IEEE International Geoscience and Remote Sensing Symposium (IGARSS).