Pavement Distress Detection with Deep Learning Using the Orthoframes Acquired by a Mobile Mapping System

The subject matter of this research article is automatic detection of pavement distress on highway roads using computer vision algorithms. Specifically, deep learning convolutional neural network models are employed towards the implementation of the detector. Source data for training the detector come in the form of orthoframes acquired by a mobile mapping system. Compared to our previous work, the orthoframes are generally of better quality, but more importantly, in this work, we introduce a manual preprocessing step: sets of orthoframes are carefully selected for training and manually digitized to ensure adequate performance of the detector. Pretrained convolutional neural networks are then fine-tuned for the problem of pavement distress detection. Corresponding experimental results are provided and analyzed and indicate a successful implementation of the detector.

[1]  Julianna M Czum,et al.  Dive Into Deep Learning. , 2020, Journal of the American College of Radiology : JACR.

[2]  Fan Yang,et al.  Feature Pyramid and Hierarchical Boosting Network for Pavement Crack Detection , 2019, IEEE Transactions on Intelligent Transportation Systems.

[3]  Gordon Morison,et al.  A Convolutional Neural Network for Pavement Surface Crack Segmentation Using Residual Connections and Attention Gating , 2019, 2019 IEEE International Conference on Image Processing (ICIP).

[4]  Yang Zhang,et al.  The Method of the Road Surface Crack Detection by the Improved Otsu Threshold , 2019, 2019 IEEE International Conference on Mechatronics and Automation (ICMA).

[5]  Andri Riid,et al.  Deep Learning for Detection of Pavement Distress using Nonideal Photographic Images , 2019, 2019 42nd International Conference on Telecommunications and Signal Processing (TSP).

[6]  Klaus Debes,et al.  Improving Visual Road Condition Assessment by Extensive Experiments on the Extended GAPs Dataset , 2019, 2019 International Joint Conference on Neural Networks (IJCNN).

[7]  George Azzopardi,et al.  Robust Inhibition-Augmented Operator for Delineation of Curvilinear Structures , 2019, IEEE Transactions on Image Processing.

[8]  Senthan Mathavan,et al.  Stereo-Based 3D Reconstruction of Potholes by a Hybrid, Dense Matching Scheme , 2019, IEEE Sensors Journal.

[9]  Bing Zhang,et al.  Intelligent Pavement Damage Monitoring Research in China , 2019, IEEE Access.

[10]  Yide Wang,et al.  A Linear Prediction and Support Vector Regression-Based Debonding Detection Method Using Step-Frequency Ground Penetrating Radar , 2019, IEEE Geoscience and Remote Sensing Letters.

[11]  Nicholay Topin,et al.  Super-convergence: very fast training of neural networks using large learning rates , 2018, Defense + Commercial Sensing.

[12]  Vishal Mandal,et al.  Automated Road Crack Detection Using Deep Convolutional Neural Networks , 2018, 2018 IEEE International Conference on Big Data (Big Data).

[13]  M. K. Osman,et al.  Crack Detection and Classification in Asphalt Pavement Images using Deep Convolution Neural Network , 2018, 2018 8th IEEE International Conference on Control System, Computing and Engineering (ICCSCE).

[14]  Kun Wang,et al.  Pavement Distress Detection Based on Transfer Learning , 2018, 2018 5th International Conference on Systems and Informatics (ICSAI).

[15]  Takahiro Tsubota,et al.  Effect of Pavement Conditions on Accident Risk in Rural Expressways , 2018, 2018 21st International Conference on Intelligent Transportation Systems (ITSC).

[16]  Nikolaos Doulamis,et al.  Combined Convolutional Neural Networks and Fuzzy Spectral Clustering for Real Time Crack Detection in Tunnels , 2018, 2018 25th IEEE International Conference on Image Processing (ICIP).

[17]  Liping Yang,et al.  Detection of Asphalt Pavement Potholes and Cracks Based on the Unmanned Aerial Vehicle Multispectral Imagery , 2018, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[18]  Anirban Chatterjee,et al.  A Fast and Accurate Automated Pavement Crack Detection Algorithm , 2018, 2018 26th European Signal Processing Conference (EUSIPCO).

[19]  Gordon Morison,et al.  A Deep Convolutional Neural Network for Semantic Pixel-Wise Segmentation of Road and Pavement Surface Cracks , 2018, 2018 26th European Signal Processing Conference (EUSIPCO).

[20]  Cédric Le Bastard,et al.  Comparative Study of Classification Algorithms to Detect Interlayer Debondings within Pavement Structures from Step-Frequency Radar Data , 2018, IGARSS 2018 - 2018 IEEE International Geoscience and Remote Sensing Symposium.

[21]  N. Pudova,et al.  Application of the Attribute Analysis for Interpretation of GPR Survey Data , 2018, 2018 17th International Conference on Ground Penetrating Radar (GPR).

[22]  Motoyuki Sato,et al.  A simplified velocity estimation method for monitoring the damaged pavement by a multistatic GPR system YAKUMO , 2018, 2018 17th International Conference on Ground Penetrating Radar (GPR).

[23]  Guohui Zhang,et al.  A Kinect-Based Approach for 3D Pavement Surface Reconstruction and Cracking Recognition , 2018, IEEE Transactions on Intelligent Transportation Systems.

[24]  Emir Buza,et al.  Pavement crack detection using Otsu thresholding for image segmentation , 2018, 2018 41st International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO).

[25]  Guiyuan Jiang,et al.  Automatic Pixel-Level Pavement Crack Detection Using Information of Multi-Scale Neighborhoods , 2018, IEEE Access.

[26]  Leslie N. Smith,et al.  A disciplined approach to neural network hyper-parameters: Part 1 - learning rate, batch size, momentum, and weight decay , 2018, ArXiv.

[27]  Sebastian Ruder,et al.  Universal Language Model Fine-tuning for Text Classification , 2018, ACL.

[28]  Atsuto Maki,et al.  A systematic study of the class imbalance problem in convolutional neural networks , 2017, Neural Networks.

[29]  Sumathi Poobal,et al.  Crack detection using image processing: A critical review and analysis , 2017, Alexandria Engineering Journal.

[30]  Mohammed Bennamoun,et al.  Cost-Sensitive Learning of Deep Feature Representations From Imbalanced Data , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[31]  Dipanjan Sarkar,et al.  Deep Learning for Computer Vision , 2018 .

[32]  Siddhartha Kumar Khaitan,et al.  Deep Convolutional Neural Networks with transfer learning for computer vision-based data-driven pavement distress detection , 2017 .

[33]  Praveen Aggarwal Predicting dynamic modulus for bituminous concrete using support vector machine , 2017, 2017 International Conference on Infocom Technologies and Unmanned Systems (Trends and Future Directions) (ICTUS).

[34]  Zhao Jiankang,et al.  Research of the algorithm calculating the length of bridge crack based on stereo vision , 2017, 2017 4th International Conference on Systems and Informatics (ICSAI).

[35]  Luca Antiga,et al.  Automatic differentiation in PyTorch , 2017 .

[36]  Khurram Kamal,et al.  Road crack detection using the particle filter , 2017, 2017 23rd International Conference on Automation and Computing (ICAC).

[37]  L. Yang,et al.  Ongoing tests and improvements of the MPS algorithm for the automatic crack detection within grey level pavement images , 2017, 2017 25th European Signal Processing Conference (EUSIPCO).

[38]  Qingquan Li,et al.  3D laser imaging and sparse points grouping for pavement crack detection , 2017, 2017 25th European Signal Processing Conference (EUSIPCO).

[39]  Zhaozheng Hu,et al.  Grid-based pavement crack analysis using deep learning , 2017, 2017 4th International Conference on Transportation Information and Safety (ICTIS).

[40]  Riccardo,et al.  Sensing road pavement health status through acoustic signals analysis , 2017, 2017 13th Conference on Ph.D. Research in Microelectronics and Electronics (PRIME).

[41]  Carmelo D'Agostino,et al.  From manual to automatic pavement distress detection and classification , 2017, 2017 5th IEEE International Conference on Models and Technologies for Intelligent Transportation Systems (MT-ITS).

[42]  Vincent Baltazart,et al.  Detection of debondings with Ground Penetrating Radar using a machine learning method , 2017, 2017 9th International Workshop on Advanced Ground Penetrating Radar (IWAGPR).

[43]  K. Tanabe,et al.  Eddy Current Testing System Using HTS-SQUID With External Pickup Coil Made of HTS Wire , 2017, IEEE Transactions on Applied Superconductivity.

[44]  Horst-Michael Groß,et al.  How to get pavement distress detection ready for deep learning? A systematic approach , 2017, 2017 International Joint Conference on Neural Networks (IJCNN).

[45]  Lina J. Karam,et al.  A Study and Comparison of Human and Deep Learning Recognition Performance under Visual Distortions , 2017, 2017 26th International Conference on Computer Communication and Networks (ICCCN).

[46]  Amir Golroo,et al.  A review on automated pavement distress detection methods , 2017 .

[47]  Ali Farhadi,et al.  YOLO9000: Better, Faster, Stronger , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[48]  Leslie N. Smith,et al.  Cyclical Learning Rates for Training Neural Networks , 2015, 2017 IEEE Winter Conference on Applications of Computer Vision (WACV).

[49]  Qingquan Li,et al.  An efficient and reliable coarse-to-fine approach for asphalt pavement crack detection , 2017, Image Vis. Comput..

[50]  Kamel Besbes,et al.  Automatic crack detection from pavement images using fuzzy thresholding , 2017, 2017 International Conference on Control, Automation and Diagnosis (ICCAD).

[51]  Qingquan Li,et al.  Path voting based pavement crack detection from laser range images , 2016, 2016 IEEE International Conference on Digital Signal Processing (DSP).

[52]  Yimin D. Zhang,et al.  Road Crack Detection Using Deep Convolutional Neural Network and Adaptive Thresholding , 2019, 2019 IEEE Intelligent Vehicles Symposium (IV).

[53]  Jiexin Pu,et al.  Research on morphological wavelet operator for crack detection of asphalt pavement , 2016, 2016 IEEE International Conference on Information and Automation (ICIA).

[54]  Zhu Qingbo,et al.  Pavement Crack Detection Algorithm Based on Image Processing Analysis , 2016, 2016 8th International Conference on Intelligent Human-Machine Systems and Cybernetics (IHMSC).

[55]  J. Choi,et al.  DETECTION OF CRACKS IN PAVED ROAD SURFACE USING LASER SCAN IMAGE DATA , 2016 .

[56]  Nima Tajbakhsh,et al.  Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning? , 2016, IEEE Transactions on Medical Imaging.

[57]  José Manuel Menéndez,et al.  A Simplified Computer Vision System for Road Surface Inspection and Maintenance , 2016, IEEE Transactions on Intelligent Transportation Systems.

[58]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[59]  Barrios A. Brayan,et al.  Pavement and base layers local thickness estimation using computer vision , 2015, 2015 10th Computing Colombian Conference (10CCC).

[60]  Patrick Donohoe,et al.  Runway assessment via remote sensing , 2015, 2015 IEEE Applied Imagery Pattern Recognition Workshop (AIPR).

[61]  Theunis F. P. Henning,et al.  Using support vector machines to predict the probability of pavement failure , 2015 .

[62]  Panos Liatsis,et al.  Optimisation of transducer positioning in air-coupled ultrasound inspection of concrete/asphalt structures , 2015, 2015 International Conference on Systems, Signals and Image Processing (IWSSIP).

[63]  Yifei Chen,et al.  Turning mobile laser scanning points into 2D/3D on-road object models: Current status , 2015, 2015 IEEE International Geoscience and Remote Sensing Symposium (IGARSS).

[64]  Ming-qing Sun,et al.  Mechanical behaviors of the thin-walled SHCC pipes under compression , 2015, 2015 International Conference on Transportation Information and Safety (ICTIS).

[65]  Andrzej Rybarczyk,et al.  Asphalt surfaced pavement cracks detection based on histograms of oriented gradients , 2015, 2015 22nd International Conference Mixed Design of Integrated Circuits & Systems (MIXDES).

[66]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[67]  Paul W. Fieguth,et al.  A review on computer vision based defect detection and condition assessment of concrete and asphalt civil infrastructure , 2015, Adv. Eng. Informatics.

[68]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[69]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[70]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[71]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[72]  Paulo Lobato Correia,et al.  Accelerated unsupervised filtering for the smoothing of road pavement surface imagery , 2014, 2014 22nd European Signal Processing Conference (EUSIPCO).

[73]  Jérôme Idier,et al.  A new minimal path selection algorithm for automatic crack detection on pavement images , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[74]  Zhenmin Tang,et al.  Pavement crack detection based on improved tensor voting , 2014, 2014 9th International Conference on Computer Science & Education.

[75]  Paulo Lobato Correia,et al.  CrackIT — An image processing toolbox for crack detection and characterization , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[76]  Eduardo Zalama Casanova,et al.  Enhanced automatic detection of road surface cracks by combining 2D/3D image processing techniques , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[77]  Cheng Wang,et al.  3D crack skeleton extraction from mobile LiDAR point clouds , 2014, 2014 IEEE Geoscience and Remote Sensing Symposium.

[78]  Eduardo Zalama Casanova,et al.  Road Crack Detection Using Visual Features Extracted by Gabor Filters , 2014, Comput. Aided Civ. Infrastructure Eng..

[79]  Li Li,et al.  AUTOMATIC PAVEMENT CRACK RECOGNITION BASED ON BP NEURAL NETWORK , 2014 .

[80]  Khurram Kamal,et al.  Metrology and visualization of potholes using the microsoft kinect sensor , 2013, 16th International IEEE Conference on Intelligent Transportation Systems (ITSC 2013).

[81]  Khurram Kamal,et al.  Pavement crack detection using the Gabor filter , 2013, 16th International IEEE Conference on Intelligent Transportation Systems (ITSC 2013).

[82]  Mohammad Hossein Fazel Zarandi,et al.  A multi-stage expert system for classification of pavement cracking , 2013, 2013 Joint IFSA World Congress and NAFIPS Annual Meeting (IFSA/NAFIPS).

[83]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[84]  E. Salari,et al.  An image-based pavement distress detection and classification , 2012, 2012 IEEE International Conference on Electro/Information Technology.

[85]  Yoshua Bengio,et al.  Deep Sparse Rectifier Neural Networks , 2011, AISTATS.

[86]  E. Salari,et al.  Pavement pothole detection and severity measurement using laser imaging , 2011, 2011 IEEE INTERNATIONAL CONFERENCE ON ELECTRO/INFORMATION TECHNOLOGY.

[87]  Wei Li,et al.  Automatic pavement cracks detection system based on Visual Studio C++ 6.0 , 2010, 2010 Sixth International Conference on Natural Computation.

[88]  Aimin Sha,et al.  Pavement crack classification based on chain code , 2010, 2010 Seventh International Conference on Fuzzy Systems and Knowledge Discovery.

[89]  W. O. Yandell,et al.  A fuzzy-control procedure for predicting fatigue crack initiation in asphaltic concrete pavements , 1994, Proceedings of 1994 IEEE 3rd International Fuzzy Systems Conference.

[90]  N. Otsu A threshold selection method from gray level histograms , 1979 .