Sewer damage detection from imbalanced CCTV inspection data using deep convolutional neural networks with hierarchical classification

Abstract Accurate infrastructure condition assessment is critical for optimized maintenance and rehabilitation plan. Closed Circuit Television (CCTV) inspection has been widely applied in the internal inspection of sewerage systems. However, the manual approach adopted under current practice is expertise intensive and time-consuming. Previous research has attempted to apply specialized image processing techniques for the detection of specific defects with engineered features, such as cracks and joint offset. However, these engineered features are less generalizable than the state-of-the-art deep learning methods. Another crucial problem in defect classification is the imbalance between defects and non-defects due to high volume of normal images and the imbalance between different defects due to varying defect occurrence rates. This raises a big challenge for both traditional methods and deep learning methods. In this paper, a method based on the deep convolutional neural network is proposed to detect and classify defects from CCTV inspections. To improve the performance on imbalanced datasets, a hierarchical classification approach is introduced to supervise the learning process at different levels. The high-level detection task tries to discriminate images with defects from normal images. The low-level classification calculates the probability of each defect assuming the image has defects. The final defect classification is then derived from the chain rule of conditional probability. The network was trained and tested using inspection images collected from 24.7 km sewer lines. The high-level defect detection accuracy was improved from 78.4% to 83.2% with a hierarchical classification approach. Due to the difficulty to discriminate the defects, the low-level defect classification accuracy still needs improvements, but the proposed network with hierarchical classification also demonstrated superior performance over traditional approaches.

[1]  Mohammed Bennamoun,et al.  Cost-Sensitive Learning of Deep Feature Representations From Imbalanced Data , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[2]  Francisco Herrera,et al.  An insight into classification with imbalanced data: Empirical results and current trends on using data intrinsic characteristics , 2013, Inf. Sci..

[3]  Ming-Der Yang,et al.  Segmenting ideal morphologies of sewer pipe defects on CCTV images for automated diagnosis , 2009, Expert Syst. Appl..

[4]  Osama Moselhi,et al.  Classification of Defects in Sewer Pipes Using Neural Networks , 2000 .

[5]  Ming-Der Yang,et al.  Automated diagnosis of sewer pipe defects based on machine learning approaches , 2008, Expert Syst. Appl..

[6]  Paul Fieguth,et al.  Automated detection of cracks in buried concrete pipe images , 2006 .

[7]  Ming-Der Yang,et al.  Morphological segmentation based on edge detection for sewer pipe defects on CCTV images , 2011, Expert Syst. Appl..

[8]  Yan Liu,et al.  Deep residual learning for image steganalysis , 2018, Multimedia Tools and Applications.

[9]  Dulcy M. Abraham,et al.  Automated defect classification in sewer closed circuit television inspections using deep convolutional neural networks , 2018, Automation in Construction.

[10]  Dulcy M. Abraham,et al.  NEURO-FUZZY APPROACHES FOR SANITARY SEWER PIPELINE CONDITION ASSESSMENT , 2001 .

[11]  Alison B. Flatau,et al.  Review Paper: Health Monitoring of Civil Infrastructure , 2003 .

[12]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Sebastian Ramos,et al.  The Cityscapes Dataset for Semantic Urban Scene Understanding , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Andrew P. Bradley,et al.  The use of the area under the ROC curve in the evaluation of machine learning algorithms , 1997, Pattern Recognit..

[15]  Yoshua Bengio,et al.  Hierarchical Probabilistic Neural Network Language Model , 2005, AISTATS.

[16]  Tarek Zayed,et al.  Automated defect detection tool for closed circuit television (cctv) inspected sewer pipelines , 2018 .

[17]  John Mashford,et al.  PIRAT—A System for Quantitative Sewer Pipe Assessment , 2000, Int. J. Robotics Res..

[18]  John Mashford,et al.  Prediction of Sewer Condition Grade Using Support Vector Machines , 2011, J. Comput. Civ. Eng..

[19]  Mohammad R. Jahanshahi,et al.  Evaluation of deep learning approaches based on convolutional neural networks for corrosion detection , 2018 .

[20]  James H. Garrett,et al.  Automated defect detection for sewer pipeline inspection and condition assessment , 2009 .

[21]  T. Davies,et al.  Sewer pipe deformation assessment by image analysis of video surveys , 1998, Pattern Recognit..

[22]  Osama Moselhi,et al.  Automated detection of surface defects in water and sewer pipes , 1999 .

[23]  Haibo He,et al.  Learning from Imbalanced Data , 2009, IEEE Transactions on Knowledge and Data Engineering.

[24]  Mahmoud R. Halfawy,et al.  Efficient Algorithm for Crack Detection in Sewer Images from Closed-Circuit Television Inspections , 2014 .

[25]  Yicheng Li,et al.  A Fast Detection Method via Region‐Based Fully Convolutional Neural Networks for Shield Tunnel Lining Defects , 2018, Comput. Aided Civ. Infrastructure Eng..

[26]  HarveyRobert Richard,et al.  Predicting the structural condition of individual sanitary sewer pipes with random forests , 2014 .

[27]  Oral Büyüköztürk,et al.  Deep Learning‐Based Crack Damage Detection Using Convolutional Neural Networks , 2017, Comput. Aided Civ. Infrastructure Eng..

[28]  Hui Han,et al.  Borderline-SMOTE: A New Over-Sampling Method in Imbalanced Data Sets Learning , 2005, ICIC.

[29]  F. Cherqui,et al.  The consistency of visual sewer inspection data , 2013 .

[30]  Andreas Geiger,et al.  Vision meets robotics: The KITTI dataset , 2013, Int. J. Robotics Res..

[31]  Robinson Piramuthu,et al.  HD-CNN: Hierarchical Deep Convolutional Neural Networks for Large Scale Visual Recognition , 2014, 2015 IEEE International Conference on Computer Vision (ICCV).

[32]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[33]  Samuel T. Ariaratnam,et al.  Innovative method for assessment of underground sewer pipe condition , 2006 .

[34]  Atsuto Maki,et al.  A systematic study of the class imbalance problem in convolutional neural networks , 2017, Neural Networks.

[35]  Chen Huang,et al.  Learning Deep Representation for Imbalanced Classification , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[36]  Mohammad R. Jahanshahi,et al.  NB-CNN: Deep Learning-Based Crack Detection Using Convolutional Neural Network and Naïve Bayes Data Fusion , 2018, IEEE Transactions on Industrial Electronics.

[37]  Dulcy M. Abraham,et al.  Assessment technologies for sewer system rehabilitation , 1998 .

[38]  Mahmoud R. Halfawy,et al.  Automated defect detection in sewer closed circuit television images using histograms of oriented gradients and support vector machine , 2014 .

[39]  Zheng Liu,et al.  State of the art review of inspection technologies for condition assessment of water pipes , 2013 .

[40]  N. Stanic,et al.  A technology for sewer pipe inspection (Part 2): Experimental assessment of a new laser profiler for sewer defect detection and quantification , 2017 .

[41]  Sunil K. Sinha,et al.  Condition assessment of underground sewer pipes using a modified digital image processing paradigm , 1999 .

[42]  Trevor Darrell,et al.  Caffe: Convolutional Architecture for Fast Feature Embedding , 2014, ACM Multimedia.

[43]  Ted DeBoda,et al.  Benefits of PACP® Version 7.0 Update NASSCO , 2015 .