A deep learning approach for real-time rebar counting on the construction site based on YOLOv3 detector

Abstract Counting steel bars is a routine daily task for steel manufacturers and most building construction sites. Currently, counting is usually performed manually, which is laborious, time-consuming, and error-prone. This study proposes a deep learning approach based on YOLOv3 detector for automatic steel bars detection and counting through images. Three new measures, including an additional feature pyramid, complete intersection over union (IoU) loss and focal loss, and bag of freebies, were introduced to improve rebar detection and counting performance. A dataset containing 74,824 rebar sections from real construction sites was constructed and utilized to evaluate the proposed approach. The application results demonstrate that the suggested measures can significantly improve the performance of the YOLOv3 detector to high average precision of 99.7% at IoU of 0.5. Comparisons with other detectors show that the proposed approach is fast, accurate, and robust at rebar counting under different construction site conditions. It can be facilitated as the basis of a real-time, cost-effective rebar counting scheme.

[1]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[2]  Yuning Jiang,et al.  UnitBox: An Advanced Object Detection Network , 2016, ACM Multimedia.

[3]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[4]  Zhaohui Zheng,et al.  Enhancing Geometric Factors in Model Learning and Inference for Object Detection and Instance Segmentation , 2020, ArXiv.

[5]  Sang Oh Park,et al.  Machine learning-based automatic reinforcing bar image analysis system in the internet of things , 2018, Multimedia Tools and Applications.

[6]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Xuefeng Zhao,et al.  Automatic pixel‐level multiple damage detection of concrete structure using fully convolutional network , 2019, Comput. Aided Civ. Infrastructure Eng..

[8]  Ali Farhadi,et al.  YOLOv3: An Incremental Improvement , 2018, ArXiv.

[9]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[10]  Cao Vu Dung,et al.  Autonomous concrete crack detection using deep fully convolutional neural network , 2019, Automation in Construction.

[11]  Hao Liu,et al.  End-Face Localization and Segmentation of Steel Bar Based on Convolution Neural Network , 2020, IEEE Access.

[12]  Ali Farhadi,et al.  YOLO9000: Better, Faster, Stronger , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[14]  Yoshua Bengio,et al.  Neural Machine Translation by Jointly Learning to Align and Translate , 2014, ICLR.

[15]  Kil To Chong,et al.  Object Detection in Very High-Resolution Aerial Images Using One-Stage Densely Connected Feature Pyramid Network , 2018, Sensors.

[16]  Zhi Xie,et al.  Bar Section Image Enhancement and Positioning Method in On-Line Steel Bar Counting and Automatic Separating System , 2008, 2008 Congress on Image and Signal Processing.

[17]  Kaiming He,et al.  Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour , 2017, ArXiv.

[18]  Ross B. Girshick,et al.  Focal Loss for Dense Object Detection , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[19]  Frank Hutter,et al.  SGDR: Stochastic Gradient Descent with Warm Restarts , 2016, ICLR.

[20]  Zhifan Feng,et al.  Rebar automatically counting on the product line , 2010, 2010 IEEE International Conference on Progress in Informatics and Computing.

[21]  Wei Liu,et al.  SSD: Single Shot MultiBox Detector , 2015, ECCV.

[22]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[23]  Tao Jiang,et al.  Automated Steel Bar Counting and Center Localization with Convolutional Neural Networks , 2019, ArXiv.

[24]  Silvio Savarese,et al.  Generalized Intersection Over Union: A Metric and a Loss for Bounding Box Regression , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Tara N. Sainath,et al.  Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups , 2012, IEEE Signal Processing Magazine.

[26]  Fan Yang,et al.  Feature Pyramid and Hierarchical Boosting Network for Pavement Crack Detection , 2019, IEEE Transactions on Intelligent Transportation Systems.

[27]  Xiaofeng Zhou,et al.  Steel bars counting and splitting method based on machine vision , 2015, 2015 IEEE International Conference on Cyber Technology in Automation, Control, and Intelligent Systems (CYBER).

[28]  Lianli Gao,et al.  Small Object Detection Using Deep Feature Pyramid Networks , 2018, PCM.

[29]  Yuan Pei-xin,et al.  Research on an Automatic Counting Method for Steel Bars' Image , 2010, 2010 International Conference on Electrical and Control Engineering.

[30]  Siddhartha Kumar Khaitan,et al.  Deep Convolutional Neural Networks with transfer learning for computer vision-based data-driven pavement distress detection , 2017 .

[31]  Ali Farhadi,et al.  You Only Look Once: Unified, Real-Time Object Detection , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  Wei Wang,et al.  Computer vision-based concrete crack detection using U-net fully convolutional networks , 2019, Automation in Construction.

[33]  Gang Yu,et al.  Cascaded Pyramid Network for Multi-person Pose Estimation , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[34]  Tjeng Wawan Cenggoro,et al.  Feature Pyramid Networks for Crowd Counting , 2019, Procedia Computer Science.

[35]  Hong-Yuan Mark Liao,et al.  YOLOv4: Optimal Speed and Accuracy of Object Detection , 2020, ArXiv.

[36]  Jixiu Wu,et al.  Automatic detection of hardhats worn by construction personnel: A deep learning approach and benchmark dataset , 2019, Automation in Construction.

[37]  Khalid M. Mosalam,et al.  Deep Transfer Learning for Image‐Based Structural Damage Recognition , 2018, Comput. Aided Civ. Infrastructure Eng..

[38]  Hui Li,et al.  Surface fatigue crack identification in steel box girder of bridges by a deep fusion convolutional neural network based on consumer-grade camera images , 2019 .

[39]  Jianyu Zhao,et al.  Design of Real-Time Steel Bars Recognition System Based on Machine Vision , 2016, 2016 8th International Conference on Intelligent Human-Machine Systems and Cybernetics (IHMSC).

[40]  Luc Van Gool,et al.  The Pascal Visual Object Classes (VOC) Challenge , 2010, International Journal of Computer Vision.

[41]  Amir H. Behzadan,et al.  Deep learning for site safety: Real-time detection of personal protective equipment , 2020 .

[42]  Zhi Zhang,et al.  Bag of Freebies for Training Object Detection Neural Networks , 2019, ArXiv.

[43]  Kaiming He,et al.  Mask R-CNN , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[44]  Pietro Perona,et al.  Microsoft COCO: Common Objects in Context , 2014, ECCV.

[45]  Oral Büyüköztürk,et al.  Deep Learning‐Based Crack Damage Detection Using Convolutional Neural Networks , 2017, Comput. Aided Civ. Infrastructure Eng..

[46]  Han Wang,et al.  Automatic Rebar Counting using Image Processing and Machine Learning , 2019, 2019 IEEE 9th Annual International Conference on CYBER Technology in Automation, Control, and Intelligent Systems (CYBER).

[47]  Hyojoo Son,et al.  Detection of construction workers under varying poses and changing background in image sequences via very deep residual networks , 2019, Automation in Construction.

[48]  Geoffrey E. Hinton,et al.  Speech recognition with deep recurrent neural networks , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[49]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[50]  Chaobo Zhang,et al.  Concrete bridge surface damage detection using a single‐stage detector , 2019, Comput. Aided Civ. Infrastructure Eng..

[51]  Kaiming He,et al.  Feature Pyramid Networks for Object Detection , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[52]  Andrew L. Maas.  Rectifier Nonlinearities Improve Neural Network Acoustic Models , 2013 .

[53]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[54]  Jian Zhang,et al.  Pixel‐level crack delineation in images with convolutional feature fusion , 2018, Structural Control and Health Monitoring.