A deep learning-based framework for an automated defect detection system for sewer pipes

Abstract The municipal drainage system is a key component of every modern city's infrastructure. However, as the drainage system ages its pipes gradually deteriorate at rates that vary based on the conditions of utilisation (i.e., intrinsic conditions) and other extrinsic factors such as the presence of trees with deep roots or the traffic load above the sewer lines, which collectively can impact the structural integrity of the pipes. As a result, regular monitoring of the drainage system is extremely important since replacement is not only costly, but, more importantly, can disturb the daily routines of citizens. In this respect, closed-circuit television (CCTV) inspection has been widely accepted as an effective inspection technology for buried infrastructure. Since sewer pipes can run for thousands of kilometers underground, cities collect massive amounts of CCTV video footage, the assessment of which is time-consuming and may require a large team of trained technologists. A framework is proposed to realize the development of a real-time automated defect detection system that takes advantage of a deep-learning algorithm. The framework focuses on streamlining the information and data flow, proposing patterns of input and output data processing. With the development of deep learning techniques, a state-of-the-art convolutional neural network (CNN) based object detector, namely YOLOv3 network, has been employed in this research. This algorithm is known to be very efficient in the field of object detection from the perspective of processing speed and accuracy. The model used in this research has been trained with a data set of 4056 samples that contains six types of defects (i.e., broken, hole, deposits, crack, fracture, and root) and one type of construction feature (tap). The performance of the model is validated with a mean average precision (mAP) of 85.37%. The proposed output of the system includes labeled CCTV videos, frames that contain defects, and associated defect information. The labeled video can serve as the benchmark for assessment technologists while the multiple output frames provide an overview of the condition of the sewer pipe.

[1]  R. A Fenner,et al.  Approaches to sewer maintenance: a review , 2000 .

[2]  Dulcy M. Abraham,et al.  Automated defect classification in sewer closed circuit television inspections using deep convolutional neural networks , 2018, Automation in Construction.

[3]  Ross B. Girshick,et al.  Fast R-CNN , 2015, 1504.08083.

[4]  Matthieu Guillaumin,et al.  Non-maximum suppression for object detection by passing messages between windows , 2015 .

[5]  Zheng Liu,et al.  State of the art review of inspection technologies for condition assessment of water pipes , 2013 .

[6]  Tae-Seong Kim,et al.  A fully integrated computer-aided diagnosis system for digital X-ray mammograms via deep learning detection, segmentation, and classification , 2018, Int. J. Medical Informatics.

[7]  Shivprakash Iyer,et al.  Segmentation of Pipe Images for Crack Detection in Buried Sewers , 2006, Comput. Aided Civ. Infrastructure Eng..

[8]  Bogdan Pavković,et al.  The Real-Time Detection of Traffic Participants Using YOLO Algorithm , 2018, 2018 26th Telecommunications Forum (TELFOR).

[9]  Jin Nakazawa,et al.  DeepCounter: Using Deep Learning to Count Garbage Bags , 2018, 2018 IEEE 24th International Conference on Embedded and Real-Time Computing Systems and Applications (RTCSA).

[10]  Ali Farhadi,et al.  YOLO9000: Better, Faster, Stronger , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[12]  Nicolás Marín,et al.  An Approach for the Automatic Classification of Work Descriptions in Construction Projects , 2015, Comput. Aided Civ. Infrastructure Eng..

[13]  Ming-Der Yang,et al.  Segmenting ideal morphologies of sewer pipe defects on CCTV images for automated diagnosis , 2009, Expert Syst. Appl..

[14]  Osama Moselhi,et al.  Automated Detection and Classification of Infiltration in Sewer Pipes , 2005 .

[15]  Kaspar Althoefer,et al.  Automated Pipe Defect Detection and Categorization Using Camera/Laser-Based Profiler and Artificial Neural Network , 2007, IEEE Transactions on Automation Science and Engineering.

[16]  James H. Garrett,et al.  Automated defect detection for sewer pipeline inspection and condition assessment , 2009 .

[17]  Mahdi Aliyari Shoorehdeli,et al.  Detection and Isolation of Interior Defects Based on Image Processing and Neural Networks: HDPE Pipeline Case Study , 2018 .

[18]  Fakhri Karray,et al.  Classification of underground pipe scanned images using feature extraction and neuro-fuzzy algorithm , 2002, IEEE Trans. Neural Networks.

[19]  Alireza Bayat,et al.  Productivity Improvement of Sewer CCTV Inspection through Time Study and Route Optimization , 2015 .

[20]  Paul Davis,et al.  An Approach Using Mathematical Morphology and Support Vector Machines to Detect Features in Pipe Images , 2008, 2008 Digital Image Computing: Techniques and Applications.

[21]  Ali Farhadi,et al.  You Only Look Once: Unified, Real-Time Object Detection , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Dulcy M. Abraham,et al.  Assessment technologies for sewer system rehabilitation , 1998 .

[23]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Sunil K. Sinha,et al.  Intelligent System for Condition Monitoring of Underground Pipelines , 2004 .

[25]  Jack Chin Pang Cheng,et al.  Automated detection of sewer pipe defects in closed-circuit television images using deep learning techniques , 2018, Automation in Construction.

[26]  Yoshua Bengio,et al.  Deep Sparse Rectifier Neural Networks , 2011, AISTATS.

[27]  Paul Fieguth,et al.  Neuro-fuzzy network for the classification of buried pipe defects , 2006 .

[28]  Luc Van Gool,et al.  The Pascal Visual Object Classes (VOC) Challenge , 2010, International Journal of Computer Vision.

[29]  Simon Jörg Sven Kirstein.  Robust adaptive flow line detection in sewer pipes , 2012 .

[30]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[31]  Osama Moselhi,et al.  Automated detection of surface defects in water and sewer pipes , 1999 .

[32]  Quoc-Lam Nguyen,et al.  Automatic recognition of asphalt pavement cracks using metaheuristic optimized edge detection algorithms and convolution neural network , 2018, Automation in Construction.

[33]  Shuai Guo,et al.  Sewer damage detection from imbalanced CCTV inspection data using deep convolutional neural networks with hierarchical classification , 2019, Automation in Construction.

[34]  Massoud Tabesh,et al.  Risk assessment model to prioritize sewer pipes inspection in wastewater collection networks. , 2017, Journal of environmental management.

[35]  Ali Farhadi,et al.  YOLOv3: An Incremental Improvement , 2018, ArXiv.

[36]  Hyeonjoon Moon,et al.  Underground sewer pipe condition assessment based on convolutional neural networks , 2019, Automation in Construction.

[37]  Wei Liu,et al.  SSD: Single Shot MultiBox Detector , 2015, ECCV.

[38]  Paul Fieguth,et al.  Computer Vision Techniques for Automatic Structural Assessment of Underground Pipes , 2003 .

[39]  Edwin Valarezo,et al.  Simultaneous Detection and Classification of Breast Masses in Digital Mammograms via a Deep Learning YOLO-based CAD System , 2018, Comput. Methods Programs Biomed..

[40]  Jian Sun,et al.  Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition , 2015, IEEE Trans. Pattern Anal. Mach. Intell..

[41]  Luc Van Gool,et al.  Efficient Non-Maximum Suppression , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[42]  Kaspar Althoefer,et al.  State of the art in sensor technologies for sewer inspection , 2002 .

[43]  Trevor Darrell,et al.  Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation , 2013, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[44]  En Li,et al.  Apple detection during different growth stages in orchards using the improved YOLO-V3 model , 2019, Comput. Electron. Agric..

[45]  Cao Vu Dung,et al.  Autonomous concrete crack detection using deep fully convolutional neural network , 2019, Automation in Construction.

[46]  E. Meeker.  The improving health of the United States, 1850-1915. , 1972, Explorations in economic history.

[47]  Samuel T. Ariaratnam,et al.  Financial Outlay Modeling for a Local Sewer Rehabilitation Strategy , 2002 .

[48]  Geoffrey E. Hinton,et al.  Rectified Linear Units Improve Restricted Boltzmann Machines , 2010, ICML.

[49]  Zheng Liu,et al.  Classification of defects with ensemble methods in the automated visual inspection of sewer pipes , 2015, Pattern Analysis and Applications.

[50]  Brajesh Dubey,et al.  A risk-based approach to sanitary sewer pipe asset management. , 2015, The Science of the total environment.

[51]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[52]  Shivprakash Iyer,et al.  A robust approach for automatic detection and segmentation of cracks in underground pipeline images , 2005, Image Vis. Comput..

[53]  K. Walsh,et al.  Deep learning for real-time fruit detection and orchard fruit load estimation: benchmarking of ‘MangoYOLO’ , 2019, Precision Agriculture.

[54]  Osama Moselhi,et al.  Classification of Defects in Sewer Pipes Using Neural Networks , 2000 .

[55]  J. Mashford,et al.  A morphological approach to pipe image interpretation based on segmentation by support vector machine , 2010 .

[56]  Ming-Der Yang,et al.  Automated diagnosis of sewer pipe defects based on machine learning approaches , 2008, Expert Syst. Appl..

[57]  Paul Fieguth,et al.  Automated detection of cracks in buried concrete pipe images , 2006 .