Computer vision for behaviour-based safety in construction: A review and future directions

Abstract The process of identifying and bringing to the fore people’s unsafe behaviour is a core function of implementing a behaviour-based safety (BBS) program in construction. This can be a labour-intensive and challenging process but is needed to enable people to reflect and learn about how their unsafe actions can jeopardise not only their safety but that of their co-workers. With advances being made in computer vision, the capability exists to automatically capture and identify unsafe behaviour and hazards in real-time from two-dimensional (2D) digital images/videos. The corollary developments in computer vision have stimulated a wealth of research in construction to examine its potential application to practice. Hindering the application of computer vision in construction has been its inability to accurately, and generalise the detection of objects. To address this shortcoming, developments in deep learning have provided computer vision with the ability to improve the accuracy, reliability and ability to generalise object detection and therefore its usage in construction. In this paper we review the developments of computer vision studies that have been used to identify unsafe behaviour from 2D images that arises on construction sites. Then, in light of advances made with deep learning, we examine and discuss its integration with computer vision to support BBS. We also suggest that future computer-vision research should aim to support BBS by being able to: (1) observe and record unsafe behaviour; (2) understand why people act unsafe behaviour; (3) learn from unsafe behaviour; and (4) predict unsafe behaviour.

[1]  Kinam Kim,et al.  Image-based construction hazard avoidance system using augmented reality in wearable device , 2017 .

[2]  Hiam Khoury,et al.  Evaluation of computer vision techniques for automated hardhat detection in indoor construction safety applications , 2018 .

[3]  Peter E. D. Love,et al.  A deep learning-based approach for mitigating falls from height with computer vision: Convolutional neural network , 2019, Adv. Eng. Informatics.

[4]  A R Duff,et al.  Contributing factors in construction accidents. , 2005, Applied ergonomics.

[5]  Hong Zhang,et al.  Development of ergonomic posture recognition technique based on 2D ordinary camera for construction hazard prevention through view-invariant features in 2D skeleton motion , 2017, Adv. Eng. Informatics.

[6]  Ali Farhadi,et al.  YOLO9000: Better, Faster, Stronger , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  SangUk Han,et al.  A vision-based motion capture and recognition framework for behavior-based safety management , 2013 .

[8]  Heng Li,et al.  Computer vision aided inspection on falling prevention measures for steeplejacks in an aerial environment , 2018, Automation in Construction.

[9]  Ronan Collobert,et al.  Learning to Segment Object Candidates , 2015, NIPS.

[10]  Ivan T. Robertson,et al.  Improving safety by the modification of behaviour , 1994 .

[11]  Peter E. D. Love,et al.  Automated detection of workers and heavy equipment on construction sites: A convolutional neural network approach , 2018, Adv. Eng. Informatics.

[12]  Feniosky Peña-Mora,et al.  Comparative Study of Motion Features for Similarity-Based Modeling and Classification of Unsafe Actions in Construction , 2014, J. Comput. Civ. Eng..

[13]  Fei Wang,et al.  Deep learning for healthcare: review, opportunities and challenges , 2018, Briefings Bioinform..

[14]  Jie Ma,et al.  Target heat-map network: An end-to-end deep network for target detection in remote sensing images , 2019, Neurocomputing.

[15]  Jürgen Schmidhuber,et al.  Deep learning in neural networks: An overview , 2014, Neural Networks.

[16]  Xiaogang Wang,et al.  Learning Feature Pyramids for Human Pose Estimation , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[17]  Weiwei Wu,et al.  An integrated information management model for proactive prevention of struck-by-falling-object accidents on construction sites , 2013 .

[18]  H. W. Heinrich,et al.  Industrial Accident Prevention: a Scientific Approach , 1951 .

[19]  J. Komaki,et al.  A behavioral approach to occupational safety: pinpointing and reinforcing safe performance in a food manufacturing plant. , 1978, The Journal of applied psychology.

[20]  Pan Zhang,et al.  Time-statistical laws of workers’ unsafe behavior in the construction industry: A case study , 2019 .

[21]  Trevor Darrell,et al.  Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation , 2013, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[22]  O. S. Williams,et al.  Causes of building construction related accident in the south-western states of Nigeria , 2019, International Journal of Built Environment and Sustainability.

[23]  C. Lawrence Zitnick,et al.  Edge Boxes: Locating Object Proposals from Edges , 2014, ECCV.

[24]  Linjia Sun.  Learning object model via segment-layout topic , 2016, 2016 IEEE 13th International Conference on Signal Processing (ICSP).

[25]  Eric Marks,et al.  Performance Test of Wireless Technologies for Personnel and Equipment Proximity Sensing in Work Zones , 2015 .

[26]  Heng Li,et al.  Proactive behavior-based safety management for construction safety improvement , 2015 .

[27]  Seokho Chi,et al.  Analyses of systems theory for construction accident prevention with specific reference to OSHA accident reports , 2013 .

[28]  P. Love,et al.  Unearthing the nature and interplay of quality and safety in construction projects: an empirical study , 2018 .

[29]  Yantao Yu,et al.  Image-and-Skeleton-Based Parameterized Approach to Real-Time Identification of Construction Workers’ Unsafe Behaviors , 2018, Journal of Construction Engineering and Management.

[30]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[31]  Peter E.D. Love,et al.  A deep hybrid learning model to detect unsafe behavior: Integrating convolution neural networks and long short-term memory , 2018 .

[32]  Vineet R. Kamat,et al.  Remote proximity monitoring between mobile construction resources using camera-mounted UAVs , 2019, Automation in Construction.

[33]  SangHyun Lee,et al.  Computer vision techniques for construction safety and health monitoring , 2015, Adv. Eng. Informatics.

[34]  Andrew Hopkins,et al.  What are we to make of safe behaviour programs , 2006 .

[35]  Peter E.D. Love,et al.  See the Difference in a Precast Facility: Changing Mindsets with an Experiential Safety Program , 2017 .

[36]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[37]  Juergen Gall,et al.  Multi-person Pose Estimation with Local Joint-to-Person Associations , 2016, ECCV Workshops.

[38]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[39]  Paul Swuste,et al.  Safety metaphors and theories, a review of the occupational safety literature of the US, UK and The Netherlands, till the first part of the 20th century , 2010 .

[40]  D. Hubel,et al.  Receptive fields, binocular interaction and functional architecture in the cat's visual cortex , 1962, The Journal of physiology.

[41]  Michael S. Lew,et al.  Deep learning for visual understanding: A review , 2016, Neurocomputing.

[42]  Xiaochun Luo,et al.  A deep learning-based method for detecting non-certified work on construction sites , 2018, Adv. Eng. Informatics.

[43]  Feniosky Peña-Mora,et al.  Vision-Based Detection of Unsafe Actions of a Construction Worker: Case Study of Ladder Climbing , 2013, J. Comput. Civ. Eng..

[44]  Simon D. Smith,et al.  Problems with safety observation reporting: A construction industry case study , 2018, Safety Science.

[45]  Jonathan Tompson,et al.  Efficient object localization using Convolutional Networks , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[46]  Ronan Collobert,et al.  Learning to Refine Object Segments , 2016, ECCV.

[47]  Jonathan Tompson,et al.  Towards Accurate Multi-person Pose Estimation in the Wild , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[48]  Kaiming He,et al.  Mask R-CNN , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[49]  Jochen Teizer,et al.  Heat map generation for predictive safety planning: preventing struck-by and near miss interactions between workers-on-foot and construction equipment , 2016 .

[50]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[51]  Koen E. A. van de Sande,et al.  Selective Search for Object Recognition , 2013, International Journal of Computer Vision.

[52]  Miroslaw J. Skibniewski,et al.  Hybrid Recommendation Approach for Behavior Modification in the Chinese Construction Industry , 2019 .

[53]  Rafael Sacks,et al.  Construction safety training using immersive virtual reality , 2013 .

[54]  David G. Lowe,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004, International Journal of Computer Vision.

[55]  Chin-Hui Lee,et al.  A unified approach to transfer learning of deep neural networks with applications to speaker adaptation in automatic speech recognition , 2016, Neurocomputing.

[56]  Robert Jenssen,et al.  Automatic autonomous vision-based power line inspection: A review of current status and the potential role of deep learning , 2018, International Journal of Electrical Power & Energy Systems.

[57]  Nikolaos Grammalidis,et al.  Dance analysis using multiple Kinect sensors , 2016, 2014 International Conference on Computer Vision Theory and Applications (VISAPP).

[58]  Rafiq M Choudhry,et al.  Behavior-based safety on construction sites: a case study. , 2014, Accident; analysis and prevention.

[59]  Peter E.D. Love,et al.  Falls from heights: A computer vision-based approach for safety harness detection , 2018, Automation in Construction.

[60]  David M. DeJoy,et al.  Behavior change versus culture change: Divergent approaches to managing workplace safety , 2005 .

[61]  Yantao Yu,et al.  An experimental study of real-time identification of construction workers' unsafe behaviors , 2017 .

[62]  Loris Nanni,et al.  Handcrafted vs. non-handcrafted features for computer vision classification , 2017, Pattern Recognit..

[63]  Xiaochun Luo,et al.  Detecting non-hardhat-use by a deep learning method from far-field surveillance videos , 2018 .

[64]  Roberto Cipolla,et al.  SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[65]  Cordelia Schmid,et al.  Action Recognition with Improved Trajectories , 2013, 2013 IEEE International Conference on Computer Vision.

[66]  Xiaochun Luo,et al.  Investigation of the causality patterns of non-helmet use behavior of construction workers , 2017 .

[67]  K. J. Seymour,et al.  Long-term evaluation of a behavior-based method for improving safety performance : a meta-analysis of 73 interrupted time-series replications , 1999 .

[68]  Ali Farhadi,et al.  You Only Look Once: Unified, Real-Time Object Detection , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[69]  Kinam Kim,et al.  Vision-Based Object-Centric Safety Assessment Using Fuzzy Inference: Monitoring Struck-By Accidents with Moving Objects , 2016, J. Comput. Civ. Eng..

[70]  Thomas S. Huang,et al.  Computer Vision: Evolution And Promise , 1996 .

[71]  Varun Ramakrishna,et al.  Convolutional Pose Machines , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[72]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[73]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[74]  Ming Yang,et al.  3D Convolutional Neural Networks for Human Action Recognition , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[75]  Dana H. Ballard,et al.  Computer Vision , 1982 .

[76]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[77]  L Y Ding,et al.  A Big-Data-based platform of workers' behavior: Observations from the field. , 2016, Accident; analysis and prevention.

[78]  Burcin Becerik-Gerber,et al.  Virtual Learning for Workers in Robot Deployed Construction Sites , 2018, Advances in Informatics and Computing in Civil and Construction Engineering.

[79]  J. Busby,et al.  Rationalizing Violation: Ordered Accounts of Intentionality in the Breaking of Safety Rules , 2016 .

[80]  Man-Woo Park,et al.  Hardhat-Wearing Detection for Enhancing On-Site Safety of Construction Workers , 2015 .