Automatic Detection and Classification of Sewer Defects via Hierarchical Deep Learning

Video and image sources are frequently applied in the area of defect inspection in industrial community. For the recognition and classification of sewer defects, a significant number of videos and images of sewers are collected. These data are then checked by human and some traditional methods to recognize and classify the sewer defects, which is inefficient and error-prone. Previously developed features like SIFT are unable to comprehensively represent such defects. Therefore, feature representation is especially important for defect autoclassification. In this paper, we study the automatic extraction of feature representation for sewer defects via deep learning. Moreover, a complete automatic system for classifying sewer defects is proposed built on a two-level hierarchical deep convolutional neural network, which shows high performance with respect to classification accuracy. The proposed network is trained on a novel data set with over 40 000 sewer images. The system has been successfully applied in the practical production, confirming its robustness and feasibility to real-world applications. The source code and trained model are available at the project website.1 Note to Practitioners—Automatic defect inspection has become a fundamental research topic in engineering application field. Specifically, sewer defect detection is an important measure for maintenance, renewal, and rehabilitation activities of sewer infrastructure. In the current operation procedure, all the captured videos need to be inspected by experts frame by frame to recognize defects, yielding a significant low inspection rate with a significant amount of time. Previous work has attempted to employ traditional image processing methods for automated sewer defect classification. However, these methods get poor generalization capabilities since they use pre-engineered features. In most cases, sewerage inspection companies have to hire numerous professional inspectors to do this job, thereby consuming a lot of human and material resources. To address this problem, the authors propose an automatic detection and classification method for sewer defects based on hierarchical deep learning. Demonstrated by various experiments, the designed framework achieves a high defect classification accuracy, which can be easily integrated into an automatic sewer defect inspection system.1https://github.com/NUAAXQ/SewerDefectDetection

[1]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[2]  Ming-Yu Liu,et al.  Deep Active Learning for Civil Infrastructure Defect Detection and Classification , 2017 .

[3]  Nikolaos Doulamis,et al.  Deep Convolutional Neural Networks for efficient vision based tunnel inspection , 2015, 2015 IEEE International Conference on Intelligent Computer Communication and Processing (ICCP).

[4]  Sergey Ioffe,et al.  Rethinking the Inception Architecture for Computer Vision , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Ming-Der Yang,et al.  Morphological segmentation based on edge detection for sewer pipe defects on CCTV images , 2011, Expert Syst. Appl..

[6]  Pei Liu,et al.  Feature Extraction of Sewer Pipe Defects Using Wavelet Transform and Co-Occurrence Matrix , 2011, Int. J. Wavelets Multiresolution Inf. Process..

[7]  Youping Chen,et al.  Classification of surface defects on steel sheet using convolutional neural networks , 2017 .

[8]  Mahmoud R. Halfawy,et al.  Integrated Vision-Based System for Automated Defect Detection in Sewer Closed Circuit Television Inspection Videos , 2015, J. Comput. Civ. Eng..

[9]  James H. Garrett,et al.  Automated defect detection for sewer pipeline inspection and condition assessment , 2009 .

[10]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[11]  Mahmoud R. Halfawy,et al.  Automated defect detection in sewer closed circuit television images using histograms of oriented gradients and support vector machine , 2014 .

[12]  Bart De Schutter,et al.  Deep convolutional neural networks for detection of rail surface defects , 2016, 2016 International Joint Conference on Neural Networks (IJCNN).

[13]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[15]  Yundong Li,et al.  Deformable Patterned Fabric Defect Detection With Fisher Criterion-Based Deep Learning , 2017, IEEE Transactions on Automation Science and Engineering.

[16]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[17]  Dongho Kang,et al.  Autonomous UAVs for Structural Health Monitoring Using Deep Learning and an Ultrasonic Beacon System with Geo‐Tagging , 2018, Comput. Aided Civ. Infrastructure Eng..

[18]  James H. Garrett,et al.  Visual Pattern Recognition Supporting Defect Reporting and Condition Assessment of Wastewater Collection Systems , 2009 .

[19]  Zheng Liu,et al.  Classification of defects with ensemble methods in the automated visual inspection of sewer pipes , 2015, Pattern Analysis and Applications.

[20]  Ali Gedam,et al.  Prediction of Sewer Pipe Main Condition Using the Linear Regression Approach , 2016 .

[21]  Wei Xu,et al.  CNN-RNN: A Unified Framework for Multi-label Image Classification , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Dulcy M. Abraham,et al.  Automated defect classification in sewer closed circuit television inspections using deep convolutional neural networks , 2018, Automation in Construction.

[23]  Trevor Darrell,et al.  Caffe: Convolutional Architecture for Fast Feature Embedding , 2014, ACM Multimedia.

[24]  Rob Fergus,et al.  Visualizing and Understanding Convolutional Networks , 2013, ECCV.

[25]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[26]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[27]  Yu-Chiang Frank Wang,et al.  Learning Deep Latent Spaces for Multi-Label Classification , 2017, ArXiv.

[28]  Mingzhu Wang,et al.  Development and Improvement of Deep Learning Based Automated Defect Detection for Sewer Pipe Inspection Using Faster R-CNN , 2018, EG-ICE.

[29]  Hichem Snoussi,et al.  A fast and robust convolutional neural network-based defect detection model in product quality control , 2017, The International Journal of Advanced Manufacturing Technology.

[30]  David G. Lowe,et al.  Object recognition from local scale-invariant features , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[31]  Rama Chellappa,et al.  Deep Multitask Learning for Railway Track Inspection , 2015, IEEE Transactions on Intelligent Transportation Systems.

[32]  J. Mashford,et al.  A morphological approach to pipe image interpretation based on segmentation by support vector machine , 2010 .

[33]  Ming-Der Yang,et al.  Automated diagnosis of sewer pipe defects based on machine learning approaches , 2008, Expert Syst. Appl..

[34]  Geoffrey E. Hinton,et al.  Rectified Linear Units Improve Restricted Boltzmann Machines , 2010, ICML.

[35]  Ming-Der Yang,et al.  Sewer pipe defects diagnosis assessment using multivariate analysis on CCTV video imagery , 2017 .

[36]  Oral Büyüköztürk,et al.  Deep Learning‐Based Crack Damage Detection Using Convolutional Neural Networks , 2017, Comput. Aided Civ. Infrastructure Eng..

[37]  Ming-Der Yang,et al.  Application of Morphological Segmentation to Leaking Defect Detection in Sewer Pipelines , 2014, Sensors.

[38]  Ngai-Man Cheung,et al.  Aircraft Fuselage Defect Detection using Deep Neural Networks , 2017, ArXiv.

[39]  Osama Moselhi,et al.  Automated Detection and Classification of Infiltration in Sewer Pipes , 2005 .

[40]  Kaspar Althoefer,et al.  Automated Pipe Defect Detection and Categorization Using Camera/Laser-Based Profiler and Artificial Neural Network , 2007, IEEE Transactions on Automation Science and Engineering.

[41]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[42]  Bernd Scholz-Reiter,et al.  Design of deep convolutional neural network architectures for automated feature extraction in industrial inspection , 2016 .

[43]  Trevor Darrell,et al.  Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation , 2013, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[44]  Mahmoud R. Halfawy,et al.  Efficient Algorithm for Crack Detection in Sewer Images from Closed-Circuit Television Inspections , 2014 .