Activity recognition of construction equipment using fractional random forest

Abstract The monitoring and tracking of construction equipment, e.g., excavators, is of great interest to improve the productivity, safety, and sustainability of construction projects. In recent years, digital technologies are leveraged to develop monitoring systems for construction equipment. These systems are commonly used to detect and/or track different pieces of equipment. However, the recent research work has indicated that the performance of the equipment monitoring system improves when they are able to also recognize/track the activities of the equipment (e.g., digging, compacting, etc.). Nevertheless, the current direction of research on equipment activity recognition is gravitating towards the use of deep learning methods. While very promising, the performance of deep learning methods is predicated on the comprehensiveness of the dataset used for training the model. Given the wide variations of construction equipment, in size and shape, the development of a comprehensive dataset can be challenging. This research hypothesizes that through the use of a robust feature augmentation method, shallow models, such as Random Forest, can yield a comparable performance without requiring a large and comprehensive dataset. Therefore, this research proposes a novel machine learning method based on the integration of Random Forest classifier with the fractional calculus-based feature augmentation technique to develop an accurate activity recognition model using a limited dataset. This method is implemented and applied to three case studies. In the first case study, the operations of two different models of excavators (one small-size and one medium-size) were tracked. By using the data from one excavator for the training and the data from the other one for testing, the impact of equipment size and operators' skill level on the performance of the proposed method is investigated. In the second case study, the data from an actual excavator was used to predict the activity of a scaled remotely controlled excavator. In the last case study, the proposed method was applied for rollers (as an example of non-articulating equipment). It is shown that the fractional feature augmentation method can have a positive impact on the performance of all machine learning methods studied in this research (i.e., Neural Network and Support Vector Machine). It is also shown that the proposed Fractional Random Forest method is able to provide comparable results to deep learning methods using considerably smaller training dataset.

[1]  Eric Marks,et al.  Method for testing proximity detection and alert technology for safe construction equipment operation , 2013 .

[2]  Xiaohui Peng,et al.  Deep Learning for Sensor-based Activity Recognition: A Survey , 2017, Pattern Recognit. Lett..

[3]  Carol C. Menassa,et al.  A vision-based marker-less pose estimation system for articulated construction robots , 2019, Automation in Construction.

[4]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[5]  David V. Anderson,et al.  Activity analysis of construction equipment using audio signals and support vector machines , 2017 .

[6]  Michele Scarpiniti,et al.  Advanced Sound Classifiers and Performance Analyses for Accurate Audio-Based Construction Project Monitoring , 2020, J. Comput. Civ. Eng..

[7]  B. Kuhlmann,et al.  IMU-based mounting parameter estimation on construction vehicles , 2017, 2017 DGON Inertial Sensors and Systems (ISS).

[8]  Amin Hammad,et al.  Dynamic equipment workspace generation for improving earthwork safety using real-time location system , 2015, Adv. Eng. Informatics.

[9]  Amin Hammad,et al.  Skeleton estimation of excavator by detecting its parts , 2017 .

[10]  R. Navon,et al.  Research in automated measurement of project performance indicators , 2007 .

[11]  Govindan Kannan,et al.  Field Studies in Construction Equipment Economics and Productivity , 2011 .

[12]  Ajibade Ayodeji Aibinu,et al.  Construction Equipment Management Practices for Improving Labor Productivity in Multistory Building Construction Projects , 2017 .

[13]  Osama Moselhi,et al.  RFID+ for Tracking Earthmoving Operations , 2012 .

[14]  Jun Wang,et al.  Two 4D Models Effective in Reducing False Alarms for Struck-by-Equipment Hazard Prevention , 2016, J. Comput. Civ. Eng..

[15]  Amir H. Behzadan,et al.  Construction equipment activity recognition for simulation input modeling using mobile sensors and machine learning classifiers , 2015, Adv. Eng. Informatics.

[16]  Amin Hammad,et al.  Risk-based look-ahead workspace generation for earthwork equipment using near real-time simulation , 2015 .

[17]  Sungwoo Moon,et al.  Effective Monitoring of the Concrete Pouring Operation in an RFID-Based Environment , 2010 .

[18]  Fujio Ikeda,et al.  A Numerical Algorithm of Discrete Fractional Calculus by using Inhomogeneous Sampling Data , 2006 .

[19]  Tuo Zhao,et al.  An enhance excavation equipments classification algorithm based on acoustic spectrum dynamic feature , 2017, Multidimens. Syst. Signal Process..

[20]  Tomohiro Yoshida,et al.  Application of RFID technology to prevention of collision accident with heavy equipment , 2010 .

[21]  Hubo Cai,et al.  Two-step long short-term memory method for identifying construction activities through positional and attentional cues , 2019, Automation in Construction.

[22]  Gary David Holt,et al.  Factors influencing construction time and cost overruns on high-rise projects in Indonesia , 1997 .

[23]  Jun Wang,et al.  Low False Alarm Rate Model for Unsafe-Proximity Detection in Construction , 2016 .

[24]  Trevor Slaton,et al.  Construction activity recognition with convolutional recurrent networks , 2020, Automation in Construction.

[25]  Juan Carlos Niebles,et al.  Vision-based action recognition of earthmoving equipment using spatio-temporal features and support vector machine classifiers , 2013, Adv. Eng. Informatics.

[26]  Daniel Rubin,et al.  Retinal Lesion Detection With Deep Learning Using Image Patches , 2018, Investigative ophthalmology & visual science.

[27]  Amin Hammad,et al.  Automated excavators activity recognition and productivity analysis from construction site surveillance videos , 2020 .

[28]  Amin Hammad,et al.  Enhancing coordination and safety of earthwork equipment operations using Multi-Agent System , 2017 .

[29]  Mahesh Pal,et al.  Random forest classifier for remote sensing classification , 2005 .

[30]  Graham W. Taylor,et al.  Dataset Augmentation in Feature Space , 2017, ICLR.

[31]  Luis F. Alarcón,et al.  Analysis of Factors Influencing Productivity Using Craftsmen Questionnaires: Case Study in a Chilean Construction Company , 2011 .

[32]  Jochen Teizer,et al.  Heat map generation for predictive safety planning: preventing struck-by and near miss interactions between workers-on-foot and construction equipment , 2016 .

[33]  Yuzhong Shen,et al.  Deep learning for effective detection of excavated soil related to illegal tunnel activities , 2017, 2017 IEEE 8th Annual Ubiquitous Computing, Electronics and Mobile Communication Conference (UEMCON).

[34]  Christophe Mues,et al.  An experimental comparison of classification algorithms for imbalanced credit scoring data sets , 2012, Expert Syst. Appl..

[35]  Apif M. Hajji,et al.  The use of construction equipment productivity rate model for estimating fuel use and carbon dioxide (CO2) emissionsCase study: bulldozer, excavator and dump truck , 2015 .

[36]  Mohd Shahir Liew,et al.  Criteria for the selection of sustainable onsite construction equipment , 2014 .

[37]  Amin Hammad,et al.  Optimization-based excavator pose estimation using real-time location systems , 2015 .

[38]  Seokho Chi,et al.  Action recognition of earthmoving excavators based on sequential pattern analysis of visual features and operation cycles , 2019, Automation in Construction.

[39]  Armin Kassemi Langroodi,et al.  Beyond data visualization: A context-realistic construction equipment training simulators , 2019, Automation in Construction.

[40]  Osama Moselhi,et al.  Productivity based method for forecasting cost & time of earthmoving operations using sampling GPS data , 2016, J. Inf. Technol. Constr..

[41]  Zhi Chen,et al.  Integrated detection and tracking of workforce and equipment from construction jobsite videos , 2017 .

[42]  Cheng Zhang,et al.  Crane Pose Estimation Using UWB Real-Time Location System , 2012, J. Comput. Civ. Eng..

[43]  Xiangyang Xue,et al.  Multi-Level Semantic Feature Augmentation for One-Shot Learning , 2018, IEEE Transactions on Image Processing.

[44]  Hao Yuan,et al.  Co-Estimation of State of Charge and State of Health for Lithium-Ion Batteries Based on Fractional-Order Calculus , 2018, IEEE Transactions on Vehicular Technology.

[45]  Yong-Cheol Lee,et al.  A Hybrid Kinematic-Acoustic System for Automated Activity Detection of Construction Equipment , 2019, Sensors.

[46]  Tao Cheng,et al.  Data Fusion of Real-Time Location Sensing and Physiological Status Monitoring for Ergonomics Analysis of Construction Workers , 2013, J. Comput. Civ. Eng..

[47]  Hyunsoo Kim,et al.  Application of dynamic time warping to the recognition of mixed equipment activities in cycle time measurement , 2018 .

[48]  Carlos H. Caldas,et al.  Learning and classifying actions of construction workers and equipment using Bag-of-Video-Feature-Words and Bayesian network models , 2011, Adv. Eng. Informatics.

[49]  Jianzhong Wang,et al.  Excavation Equipment Recognition Based on Novel Acoustic Statistical Features , 2017, IEEE Transactions on Cybernetics.

[50]  Christopher M. Bishop,et al.  Pattern Recognition and Machine Learning (Information Science and Statistics) , 2006 .

[51]  Jochen Teizer,et al.  Automatic spatio-temporal analysis of construction site equipment operations using GPS data , 2013 .

[52]  Khandakar M. Rashid,et al.  Times-series data augmentation and deep learning for construction equipment activity recognition , 2019, Adv. Eng. Informatics.

[53]  Hyoungkwan Kim,et al.  Using Hue, Saturation, and Value Color Space for Hydraulic Excavator Idle Time Analysis , 2007 .

[54]  Jie Gu,et al.  An effective intrusion detection framework based on SVM with feature augmentation , 2017, Knowl. Based Syst..

[55]  Onur Behzat Tokdemir,et al.  Predicting the outcome of construction incidents , 2019, Safety Science.

[56]  Tin Kam Ho,et al.  The Random Subspace Method for Constructing Decision Forests , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[57]  David V. Anderson,et al.  Audio Signal Processing for Activity Recognition of Construction Heavy Equipment , 2016 .

[58]  Wei Zhao,et al.  Research on the deep learning of the small sample data based on transfer learning , 2017 .

[59]  Yang Miang Goh,et al.  Safety leading indicators for construction sites: A machine learning approach , 2018, Automation in Construction.

[60]  Soungho Chae Development of Warning System for Preventing Collision Accident on Construction Site , 2009 .

[61]  Guoyin Wang,et al.  Complete Random Forest Based Class Noise Filtering Learning for Improving the Generalizability of Classifiers , 2019, IEEE Transactions on Knowledge and Data Engineering.

[62]  Aske Plaat,et al.  On the Impact of Data Set Size in Transfer Learning Using Deep Neural Networks , 2016, IDA.

[63]  Fernanda Leite,et al.  Exploring approaches to improve the performance of autonomous monitoring with imperfect data in location-aware wireless sensor networks , 2014, Adv. Eng. Informatics.

[64]  Rabha W. Ibrahim The Fractional Differential Polynomial Neural Network for Approximation of Functions , 2013, Entropy.

[65]  Sven J. Dickinson,et al.  Server-Customer Interaction Tracker: Computer Vision-Based System to Estimate Dirt-Loading Cycles , 2013 .

[66]  William Rasdorf,et al.  Impact of Engine Idling on Fuel Use and CO2 Emissions of Nonroad Diesel Construction Equipment , 2012 .

[67]  Seokho Chi,et al.  Interaction analysis for vision-based activity identification of earthmoving excavators and dump trucks , 2018 .

[68]  Chuang Li,et al.  Estimation of the nitrogen concentration of rubber tree using fractional calculus augmented NIR spectra , 2017 .

[69]  Jochen Teizer,et al.  Visibility-related fatalities related to construction equipment , 2011 .

[70]  Yacine Rezgui,et al.  Trees vs Neurons: Comparison between random forest and ANN for high-resolution prediction of building energy consumption , 2017 .

[71]  Seokho Chi,et al.  Automated Object Identification Using Optical Video Cameras on Construction Sites , 2011, Comput. Aided Civ. Infrastructure Eng..

[72]  Q. Han,et al.  A generalized method to predict the compressive strength of high-performance concrete by improved random forest algorithm , 2019, Construction and Building Materials.

[73]  Amin Hammad,et al.  Framework for near real-time simulation of earthmoving projects using location tracking technologies , 2014 .

[74]  Brenda McCabe,et al.  Part based model and spatial–temporal reasoning to recognize hydraulic excavators in construction images and videos , 2012 .

[75]  Mumtaz A. Usmen,et al.  Worker safety and injury severity analysis of earthmoving equipment accidents. , 2018, Journal of safety research.

[76]  Brenda McCabe,et al.  Automated Visual Recognition of Dump Trucks in Construction Videos , 2012, J. Comput. Civ. Eng..

[77]  S. Das,et al.  Functional Fractional Calculus for System Identification and Controls , 2007 .

[78]  Tao Cheng,et al.  Proximity hazard indicator for workers-on-foot near miss interactions with construction equipment and geo-referenced hazard areas , 2015 .

[79]  Jimmie Hinze,et al.  The Nature of Struck-by Accidents , 2005 .

[80]  Tao Cheng,et al.  Real-time resource location data collection and visualization technology for construction safety and activity monitoring applications , 2013 .

[81]  Sunil K. Sinha,et al.  Construction equipment productivity estimation using artificial neural network model , 2006 .

[82]  Mani Golparvar-Fard,et al.  Automated 2D detection of construction equipment and workers from site video streams using histograms of oriented gradients and colors , 2013 .

[83]  Vineet R. Kamat,et al.  Optical marker‐based end effector pose estimation for articulated excavators , 2016 .

[84]  Cheng Zhang,et al.  Multiagent Approach for Real-Time Collision Avoidance and Path Replanning for Cranes , 2012, J. Comput. Civ. Eng..

[85]  Amir H. Behzadan,et al.  An integrated data collection and analysis framework for remote monitoring and planning of construction operations , 2012, Adv. Eng. Informatics.

[86]  A. Prasad,et al.  Newer Classification and Regression Tree Techniques: Bagging and Random Forests for Ecological Prediction , 2006, Ecosystems.

[87]  Hwan-Sik Yoon,et al.  Vision-based estimation of excavator manipulator pose for automated grading control , 2019, Automation in Construction.

[88]  Jimmie Hinze,et al.  Autonomous pro-active real-time construction worker and equipment operator proximity safety alert system , 2010 .

[89]  Jianzhong Wang,et al.  Excavation equipment classification based on improved MFCC features and ELM , 2017, Neurocomputing.

[90]  Amir H. Behzadan,et al.  Knowledge-Based Simulation Modeling of Construction Fleet Operations Using Multimodal-Process Data Mining , 2013 .

[91]  Shamil Naoum,et al.  Factors influencing labor productivity on construction sites , 2016 .

[92]  Mani Golparvar-Fard,et al.  Automated Methods for Activity Recognition of Construction Workers and Equipment: State-of-the-Art Review , 2020 .