DEEVA: A Deep Learning and IoT Based Computer Vision System to Address Safety and Security of Production Sites in Energy Industry

When it comes to addressing the safety/security related needs at different production/construction sites, accurate detection of the presence of workers, vehicles, equipment important and formed an integral part of computer vision-based surveillance systems (CVSS). Traditional CVSS systems focus on the use of different computer vision and pattern recognition algorithms overly reliant on manual extraction of features and small datasets, limiting their usage because of low accuracy, need for expert knowledge and high computational costs. The main objective of this paper is to provide decision makers at sites with a practical yet comprehensive deep learning and IoT based solution to tackle various computer vision related problems such as scene classification, object detection in scenes, semantic segmentation, scene captioning etc. Our overarching goal is to address the central question of What is happening at this site and where is it happening in an automated fashion minimizing the need for human resources dedicated to surveillance. We developed Deep ExxonMobil Eye for Video Analysis (DEEVA) package to handle scene classification, object detection, semantic segmentation and captioning of scenes in a hierarchical approach. The results reveal that transfer learning with the RetinaNet object detector is able to detect the presence of workers, different types of vehicles/construction equipment, safety related objects at a high level of accuracy (above 90%). With the help of deep learning to automatically extract features and IoT technology to automatic capture, transfer and process vast amount of realtime images, this framework is an important step towards the development of intelligent surveillance systems aimed at addressing myriads of open ended problems in the realm of security/safety monitoring, productivity assessments and future decision making.

[1]  Hyeran Byun,et al.  Detecting Construction Equipment Using a Region-Based Fully Convolutional Network and Transfer Learning , 2018, J. Comput. Civ. Eng..

[2]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Wei Liu,et al.  SSD: Single Shot MultiBox Detector , 2015, ECCV.

[4]  Jordi Pont-Tuset,et al.  The Open Images Dataset V4 , 2018, International Journal of Computer Vision.

[5]  Ross B. Girshick,et al.  Focal Loss for Dense Object Detection , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[6]  Andreas Geiger,et al.  Are we ready for autonomous driving? The KITTI vision benchmark suite , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[7]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[8]  Ross B. Girshick,et al.  Fast R-CNN , 2015, 1504.08083.

[9]  Peter E. D. Love,et al.  Automated detection of workers and heavy equipment on construction sites: A convolutional neural network approach , 2018, Adv. Eng. Informatics.

[10]  Sergio Guadarrama,et al.  Speed/Accuracy Trade-Offs for Modern Convolutional Object Detectors , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Pietro Perona,et al.  Microsoft COCO: Common Objects in Context , 2014, ECCV.

[12]  Zhiming Luo,et al.  MIO-TCD: A New Benchmark Dataset for Vehicle Classification and Localization , 2018, IEEE Transactions on Image Processing.

[13]  Ali Farhadi,et al.  You Only Look Once: Unified, Real-Time Object Detection , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Sebastian Ramos,et al.  The Cityscapes Dataset for Semantic Urban Scene Understanding , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Peter E.D. Love,et al.  A deep hybrid learning model to detect unsafe behavior: Integrating convolution neural networks and long short-term memory , 2018 .

[16]  Kaiming He,et al.  Feature Pyramid Networks for Object Detection , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Jixiu Wu,et al.  Automatic detection of hardhats worn by construction personnel: A deep learning approach and benchmark dataset , 2019, Automation in Construction.