Evaluating the Work Productivity of Assembling Reinforcement through the Objects Detected by Deep Learning

With the rapid development of deep learning, computer vision has assisted in solving a variety of problems in engineering construction. However, very few computer vision-based approaches have been proposed on work productivity’s evaluation. Therefore, taking a super high-rise project as a research case, using the detected object information obtained by a deep learning algorithm, a computer vision-based method for evaluating the productivity of assembling reinforcement is proposed. Firstly, a detector that can accurately distinguish various entities related to assembling reinforcement based on CenterNet is established. DLA34 is selected as the backbone. The mAP reaches 0.9682, and the speed of detecting a single image can be as low as 0.076 s. Secondly, the trained detector is used to detect the video frames, and images with detected boxes and documents with coordinates can be obtained. The position relationship between the detected work objects and detected workers is used to determine how many workers (N) have participated in the task. The time (T) to perform the process can be obtained from the change of coordinates of the work object. Finally, the productivity is evaluated according to N and T. The authors use four actual construction videos for validation, and the results show that the productivity evaluation is generally consistent with the actual conditions. The contribution of this research to construction management is twofold: On the one hand, without affecting the normal behavior of workers, a connection between construction individuals and work object is established, and the work productivity evaluation is realized. On the other hand, the proposed method has a positive effect on improving the efficiency of construction management.

[1]  Jianyu Yang,et al.  Ship Detection in Large-Scale SAR Images Via Spatial Shuffle-Group Enhance Attention , 2021, IEEE Trans. Geosci. Remote. Sens..

[2]  Trevor Slaton,et al.  Construction activity recognition with convolutional recurrent networks , 2020, Automation in Construction.

[3]  Vineet R. Kamat,et al.  Remote proximity monitoring between mobile construction resources using camera-mounted UAVs , 2019, Automation in Construction.

[4]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[5]  Roger Ghanem,et al.  A novel approach for the structural identification and monitoring of a full-scale 17-story building based on ambient vibration measurements , 2008 .

[6]  Vinod Vokkarane,et al.  DeepFood: Deep Learning-Based Food Image Recognition for Computer-Aided Dietary Assessment , 2016, ICOST.

[7]  Di He,et al.  Defect detection of hot rolled steels with a new object detection framework called classification priority network , 2019, Comput. Ind. Eng..

[8]  Gwanggil Jeon,et al.  Edge computing-based person detection system for top view surveillance: Using CenterNet with transfer learning , 2021, Appl. Soft Comput..

[9]  Xiaochun Luo,et al.  Recognizing Diverse Construction Activities in Site Images via Relevance Networks of Construction-Related Objects Detected by Convolutional Neural Networks , 2018, J. Comput. Civ. Eng..

[10]  Ping He,et al.  Manifesting construction activity scenes via image captioning , 2020 .

[11]  Niannian Wang,et al.  Bolt loosening angle detection technology using deep learning , 2018, Structural Control and Health Monitoring.

[12]  Wei Liu,et al.  SSD: Single Shot MultiBox Detector , 2015, ECCV.

[13]  Xiaochun Luo,et al.  Towards efficient and objective work sampling: Recognizing workers' activities in site surveillance videos with two-stream convolutional networks , 2018, Automation in Construction.

[14]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[15]  Oral Büyüköztürk,et al.  Autonomous Structural Visual Inspection Using Region‐Based Deep Learning for Detecting Multiple Damage Types , 2018, Comput. Aided Civ. Infrastructure Eng..

[16]  Hui Li,et al.  Automatic seismic damage identification of reinforced concrete columns from images by a region‐based deep convolutional neural network , 2019, Structural Control and Health Monitoring.

[17]  Xiaochun Luo,et al.  Capturing and Understanding Workers’ Activities in Far‐Field Surveillance Videos with Deep Action Recognition and Bayesian Nonparametric Learning , 2018, Comput. Aided Civ. Infrastructure Eng..

[18]  Jun Chen,et al.  A deep learning approach for real-time rebar counting on the construction site based on YOLOv3 detector , 2021 .

[19]  Peng Zhao,et al.  CNN-based statistics and location estimation of missing components in routine inspection of historic buildings , 2019, Journal of Cultural Heritage.

[20]  Sebastian Thrun,et al.  Dermatologist-level classification of skin cancer with deep neural networks , 2017, Nature.

[21]  Wensheng Su,et al.  Autonomous bolt loosening detection using deep learning , 2019, Structural Health Monitoring.

[22]  Young-Jin Cha,et al.  SDDNet: Real-Time Crack Segmentation , 2020, IEEE Transactions on Industrial Electronics.

[23]  Houtan Jebelli,et al.  Automated Action Recognition Using an Accelerometer-Embedded Wristband-Type Activity Tracker , 2019, Journal of Construction Engineering and Management.

[24]  Hei Law,et al.  CornerNet: Detecting Objects as Paired Keypoints , 2018, ECCV.

[25]  Jinping Ou,et al.  Shaking table tests for evaluating the damage features under earthquake excitations using smartphones , 2018, Smart Structures and Materials + Nondestructive Evaluation and Health Monitoring.

[26]  Peter E.D. Love,et al.  Mapping computer vision research in construction: Developments, knowledge gaps and implications for research , 2019, Automation in Construction.

[27]  Peter E.D. Love,et al.  Falls from heights: A computer vision-based approach for safety harness detection , 2018, Automation in Construction.

[28]  Hong-wei Huang,et al.  Quick image analysis of concrete pore structure based on deep learning , 2019, Construction and Building Materials.

[29]  Gang Li,et al.  Automatic bridge crack identification from concrete surface using ResNeXt with postprocessing , 2020, Structural Control and Health Monitoring.

[30]  Xiaochun Luo,et al.  Motion-based analysis for construction workers using biomechanical methods , 2017 .

[31]  Ali Farhadi,et al.  You Only Look Once: Unified, Real-Time Object Detection , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  Lan Liu,et al.  Efficient Foreign Object Detection Between PSDs and Metro Doors via Deep Neural Networks , 2020, IEEE Access.

[33]  Mingyuan Zhang,et al.  Safety Distance Identification for Crane Drivers Based on Mask R-CNN , 2019, Sensors.

[34]  Naif Alajlan,et al.  Deep Learning Approach for Car Detection in UAV Imagery , 2017, Remote. Sens..

[35]  Timothy Bretl,et al.  Detecting and Classifying Cranes Using Camera-Equipped UAVs for Monitoring Crane-Related Safety Hazards , 2017 .

[36]  Hayder Radha,et al.  Deep learning algorithm for autonomous driving using GoogLeNet , 2017, 2017 IEEE Intelligent Vehicles Symposium (IV).

[37]  Wearable sensor-based activity recognition for data-driven simulation of construction workers' activities , 2015, WSC 2015.

[38]  Mohsin M. Jamali,et al.  Multi-Object Detection and Tracking, Based on DNN, for Autonomous Vehicles: A Review , 2021, IEEE Sensors Journal.

[39]  Xuefeng Zhao,et al.  Automatic Crack Detection and Measurement of Concrete Structure Using Convolutional Encoder-Decoder Network , 2020, IEEE Access.

[40]  Mani Golparvar-Fard,et al.  Human-object interaction recognition for automatic construction site safety inspection , 2020 .

[41]  Zhen Yang,et al.  Research on Construction Workers’ Activity Recognition Based on Smartphone , 2018, Sensors.

[42]  Amir H. Behzadan,et al.  Wearable sensor-based activity recognition for data-driven simulation of construction workers' activities , 2015, 2015 Winter Simulation Conference (WSC).

[43]  Р Ю Чуйков,et al.  Обнаружение транспортных средств на изображениях загородных шоссе на основе метода Single shot multibox Detector , 2017 .

[44]  Keiji Yanai,et al.  Food image recognition using deep convolutional network with pre-training and fine-tuning , 2015, 2015 IEEE International Conference on Multimedia & Expo Workshops (ICMEW).

[45]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[46]  Moncef Gabbouj,et al.  Real-time vibration-based structural damage detection using one-dimensional convolutional neural networks , 2017 .

[47]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[48]  Ross B. Girshick,et al.  Mask R-CNN , 2017, 1703.06870.

[49]  Mingchu Li,et al.  Smartphone-Based Mobile Testing Technique for Quick Bridge Cable–Force Measurement , 2017 .

[50]  Dimos Polyzois,et al.  Deep learning-based automatic volumetric damage quantification using depth camera , 2019, Automation in Construction.

[51]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[52]  Jing Wang,et al.  Automatic Recognition of Highway Tunnel Defects Based on an Improved U-Net Model , 2019, IEEE Sensors Journal.

[53]  Bo Chen,et al.  MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications , 2017, ArXiv.

[54]  Ali Farhadi,et al.  YOLO9000: Better, Faster, Stronger , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[55]  Mark Sandler,et al.  MobileNetV2: Inverted Residuals and Linear Bottlenecks , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[56]  Xingyi Zhou,et al.  Objects as Points , 2019, ArXiv.

[57]  Yantao Yu,et al.  An experimental study of real-time identification of construction workers' unsafe behaviors , 2017 .

[58]  Hassan Mathkour,et al.  Towards Deep Object Detection Techniques for Phoneme Recognition , 2020, IEEE Access.

[59]  SangUk Han,et al.  A vision-based motion capture and recognition framework for behavior-based safety management , 2013 .

[60]  Hubo Cai,et al.  Two-step long short-term memory method for identifying construction activities through positional and attentional cues , 2019, Automation in Construction.

[61]  Bram van Ginneken,et al.  Pulmonary Nodule Detection in CT Images: False Positive Reduction Using Multi-View Convolutional Networks , 2016, IEEE Transactions on Medical Imaging.

[62]  Wenjin Tao,et al.  Worker Activity Recognition in Smart Manufacturing Using IMU and sEMG Signals with Convolutional Neural Networks , 2018, EasyChair Preprints.

[63]  Rob Fergus,et al.  Visualizing and Understanding Convolutional Networks , 2013, ECCV.

[64]  Hong-Yuan Mark Liao,et al.  YOLOv4: Optimal Speed and Accuracy of Object Detection , 2020, ArXiv.

[65]  Xuefeng Zhao,et al.  Method for detecting road pavement damage based on deep learning , 2019, Smart Structures and Materials + Nondestructive Evaluation and Health Monitoring.

[66]  Jixiu Wu,et al.  Automatic detection of hardhats worn by construction personnel: A deep learning approach and benchmark dataset , 2019, Automation in Construction.

[67]  Zhongke Shi,et al.  Vision-Based Tower Crane Tracking for Understanding Construction Activity , 2014, J. Comput. Civ. Eng..

[68]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[69]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[70]  Ali Farhadi,et al.  YOLOv3: An Incremental Improvement , 2018, ArXiv.

[71]  Xiaochun Luo,et al.  Detecting non-hardhat-use by a deep learning method from far-field surveillance videos , 2018 .

[72]  Dongho Kang,et al.  Autonomous UAVs for Structural Health Monitoring Using Deep Learning and an Ultrasonic Beacon System with Geo‐Tagging , 2018, Comput. Aided Civ. Infrastructure Eng..

[73]  Chinmay Chakraborty,et al.  Grape Disease Detection Network Based on Multi-Task Learning and Attention Features , 2021, IEEE Sensors Journal.

[74]  Man-Woo Park,et al.  Hardhat-Wearing Detection for Enhancing On-Site Safety of Construction Workers , 2015 .

[75]  Subhashini Venugopalan,et al.  Development and Validation of a Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus Photographs. , 2016, JAMA.

[76]  Yang Liu,et al.  Automated Pixel‐Level Pavement Crack Detection on 3D Asphalt Surfaces Using a Deep‐Learning Network , 2017, Comput. Aided Civ. Infrastructure Eng..