Vision-based nonintrusive context documentation for earthmoving productivity simulation

Abstract Although video surveillance systems have shown potential for analyzing jobsite contexts, the necessity of a complex multi-camera surveillance system or workers' privacy issues remain as substantive hurdles to adopt such systems in practice. To address such issues, this study presents a non-intrusive earthmoving productivity analysis method using imaging and simulation. The site access log of dump trucks is used to infer earthmoving contexts, which is produced by analyzing videos recorded at the entrance and the exit of a construction site. An algorithm for license plate detection and recognition in an uncontrolled environment is developed to automatically produce the site access log, by leveraging video deinterlacing, a deep convolutional network, and rule-based post-processing. The experimental results show the effectiveness of the proposed method for producing the site access log. Based on the site access log, simulation-based productivity analysis is conducted to produce a daily productivity report, which can provide the basis for earthmoving resource planning. It is expected that the resulting daily productivity report promotes data-driven decision-making for earthmoving resource allocation, thereby improving potential for saving cost and time for earthworks with an updated resource allocation plan.

[1]  Zhongke Shi,et al.  Vision-based action recognition of construction workers using dense trajectories , 2016, Adv. Eng. Informatics.

[2]  Hyunsoo Kim,et al.  Application of dynamic time warping to the recognition of mixed equipment activities in cycle time measurement , 2018 .

[3]  Eddy M. Rojas,et al.  Estimation of Optimal Productivity in Labor-Intensive Construction Operations: Advanced Study , 2018, Journal of Construction Engineering and Management.

[4]  Hong Zhang.  Multi-objective simulation-optimization for earthmoving operations , 2008 .

[5]  Daniel W. Halpin,et al.  CYCLONE — Method for Modeling Job Site Processes , 1977 .

[6]  AlshibaniAdel,et al.  Estimating productivity of earthmoving operations using spatial technologies11This paper is one of a selection of papers in this Special Issue on Construction Engineering and Management. , 2012 .

[7]  Amir H. Behzadan,et al.  Construction equipment activity recognition for simulation input modeling using mobile sensors and machine learning classifiers , 2015, Adv. Eng. Informatics.

[8]  K. Ball.  Workplace surveillance: an overview , 2010 .

[9]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[10]  Jie Gong,et al.  Computer Vision-Based Video Interpretation Model for Automated Productivity Analysis of Construction Operations , 2010 .

[11]  Seokho Chi,et al.  Image-Based Safety Assessment: Automated Spatial Safety Risk Identification of Earthmoving and Surface Mining Activities , 2012 .

[12]  Simaan M. AbouRizk,et al.  Research in Modeling and Simulation for Improving Construction Engineering Operations , 2011 .

[13]  Hong-Hyun Kim,et al.  Multi-task convolutional neural network system for license plate recognition , 2017, International Journal of Control, Automation and Systems.

[14]  Lianwen Jin,et al.  A New CNN-Based Method for Multi-Directional Car License Plate Detection , 2018, IEEE Transactions on Intelligent Transportation Systems.

[15]  Mani Golparvar-Fard,et al.  Crowdsourcing Construction Activity Analysis from Jobsite Video Streams , 2015 .

[16]  Trevor Darrell,et al.  Caffe: Convolutional Architecture for Fast Feature Embedding , 2014, ACM Multimedia.

[17]  Han-Seong Gwak,et al.  Optimal cut-fill pairing and sequencing method in earthwork operation , 2018 .

[18]  Juan Carlos Niebles,et al.  Vision-based action recognition of earthmoving equipment using spatio-temporal features and support vector machine classifiers , 2013, Adv. Eng. Informatics.

[19]  Simaan M. AbouRizk,et al.  Identification of Invariant Average Weighted Haul Distance to Simplify Earthmoving Simulation Modeling in Planning Site Grading Operations , 2014 .

[20]  Simaan M. AbouRizk,et al.  Role of Simulation in Construction Engineering and Management , 2010 .

[21]  Hyeran Byun,et al.  Detecting Construction Equipment Using a Region-Based Fully Convolutional Network and Transfer Learning , 2018, J. Comput. Civ. Eng..

[22]  Giovanni C. Migliaccio,et al.  Temporal Effect of Construction Workforce Physical Strain on Diminishing Marginal Productivity at the Task Level , 2018, Journal of Construction Engineering and Management.

[23]  Krzysztof Schabowicz,et al.  Estimation of earthworks execution time cost by means of artificial neural networks , 2010 .

[24]  Ali Khaloo,et al.  Unmanned aerial vehicle inspection of the Placer River Trail Bridge through image-based 3D modelling , 2018 .

[25]  Xiaochun Luo,et al.  Detecting non-hardhat-use by a deep learning method from far-field surveillance videos , 2018 .

[26]  Patricio A. Vela,et al.  Fusion of Photogrammetry and Video Analysis for Productivity Assessment of Earthwork Processes , 2017, Comput. Aided Civ. Infrastructure Eng..

[27]  Jeffrey S. Russell,et al.  Framework for an intelligent earthwork system: Part II. Task identification/scheduling and resource allocation methodology , 2003 .

[28]  Jeffrey S. Russell,et al.  Framework for an intelligent earthwork system: Part I. System architecture , 2003 .

[29]  Ehsan Rezazadeh Azar.  Semantic Annotation of Videos from Equipment-Intensive Construction Operations by Shot Recognition and Probabilistic Reasoning , 2017 .

[30]  Osama Moselhi,et al.  Estimating productivity of earthmoving operations using spatial technologies 1 , 2012 .

[31]  Youngjib Ham,et al.  Analyzing context and productivity of tunnel earthmoving processes using imaging and simulation , 2018, Automation in Construction.

[32]  Dong-Eun Lee,et al.  Stochastic dozer productivity estimation method , 2017 .

[33]  Seung-Woo Han,et al.  Technical comparisons of simulation-based productivity prediction methodologies by means of estimation tools focusing on conventional earthmovings , 2011 .

[34]  Brenda McCabe,et al.  Automated computer vision-based detection of components of under-construction indoor partitions , 2017 .

[35]  Ehsan Rezazadeh Azar.  Construction Equipment Identification Using Marker-Based Recognition and an Active Zoom Camera , 2016 .

[36]  Jong Won Seo,et al.  3D-GIS Based Earthwork Planning System for Productivity Improvement , 2009 .

[37]  Man-Woo Park,et al.  Hardhat-Wearing Detection for Enhancing On-Site Safety of Construction Workers , 2015 .

[38]  Shahab A. Abdulla,et al.  Ensemble of adaboost cascades of 3L-LBPs classifiers for license plates detection with low quality images , 2018, Expert Syst. Appl..

[39]  Christian Koch,et al.  Automated Pothole Distress Assessment Using Asphalt Pavement Video Data , 2013 .

[40]  Peter E.D. Love,et al.  Falls from heights: A computer vision-based approach for safety harness detection , 2018, Automation in Construction.

[41]  Patricio A. Vela,et al.  Construction performance monitoring via still images, time-lapse photos, and video streams: Now, tomorrow, and the future , 2015, Adv. Eng. Informatics.

[42]  Lingguang Song,et al.  Adaptive real-time tracking and simulation of heavy construction operations for look-ahead scheduling , 2012 .

[43]  Kinam Kim,et al.  Vision-Based Object-Centric Safety Assessment Using Fuzzy Inference: Monitoring Struck-By Accidents with Moving Objects , 2016, J. Comput. Civ. Eng..

[44]  Arindam Gan Chowdhury,et al.  Imaging-to-Simulation Framework for Improving Disaster Preparedness of Construction Projects and Neighboring Communities , 2017 .

[45]  SangHyun Lee,et al.  Computer vision techniques for construction safety and health monitoring , 2015, Adv. Eng. Informatics.

[46]  Jongwon Seo,et al.  Utilization analysis methodology for fleet telematics of heavy earthwork equipment , 2018, Automation in Construction.

[47]  Daniel Mora-Meliá,et al.  Weather-wise: A weather-aware planning tool for improving construction productivity and dealing with claims , 2017 .

[48]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[49]  Amin Hammad,et al.  Framework for near real-time simulation of earthmoving projects using location tracking technologies , 2014 .

[50]  Burcu Akinci,et al.  Proactive Productivity Management at Job Sites: Understanding Characteristics of Assumptions Made for Construction Processes during Planning Based on Case Studies and Interviews , 2014 .

[51]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[52]  Amin Hammad,et al.  Predicting movements of onsite workers and mobile equipment for enhancing construction site safety , 2016 .

[53]  Seung-Hyun Lee,et al.  Development of scheduling model for earth work using genetic algorithm , 2014 .

[54]  Peter E.D. Love,et al.  A deep hybrid learning model to detect unsafe behavior: Integrating convolution neural networks and long short-term memory , 2018 .

[55]  Kinam Kim,et al.  Data-driven scene parsing method for recognizing construction site objects in the whole image , 2016 .

[56]  Xiaogang Wang,et al.  Intelligent multi-camera video surveillance: A review , 2013, Pattern Recognit. Lett..

[57]  Sven J. Dickinson,et al.  Server-Customer Interaction Tracker: Computer Vision-Based System to Estimate Dirt-Loading Cycles , 2013 .

[58]  Ling Shao,et al.  Transfer Learning for Visual Categorization: A Survey , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[59]  S. Abourizk,et al.  STATISTICAL PROPERTIES OF CONSTRUCTION DURATION DATA , 1992 .

[60]  Seokho Chi,et al.  Interaction analysis for vision-based activity identification of earthmoving excavators and dump trucks , 2018 .

[61]  Changmin Kim,et al.  Classification of major construction materials in construction environments using ensemble classifiers , 2014, Adv. Eng. Informatics.

[62]  Osama Moselhi,et al.  Truck+ for earthmoving operations , 2014, J. Inf. Technol. Constr..

[63]  Eddy M. Rojas,et al.  Is Construction Labor Productivity Really Declining , 2003 .

[64]  Mani Golparvar-Fard,et al.  Appearance-based material classification for monitoring of operation-level construction progress using 4D BIM and site photologs , 2015 .

[65]  Paul M. Goodrum,et al.  Validation Methodologies and Their Impact in Construction Productivity Research , 2014 .

[66]  António Gomes Correia,et al.  An evolutionary multi-objective optimization system for earthworks , 2015, Expert Syst. Appl..

[67]  Chang Liu,et al.  Optimizing Earthmoving Job Planning Based on Evaluation of Temporary Haul Road Networks Design for Mass Earthworks Projects , 2015 .

[68]  Yong Zhao,et al.  License plate detection based on fully convolutional networks , 2017, J. Electronic Imaging.