SuPEr-SAM: Using the Supervision Signal from a Pose Estimator to Train a Spatial Attention Module for Personal Protective Equipment Recognition

We propose a deep learning method to automatically detect personal protective equipment (PPE), such as helmets, surgical masks, reflective vests, boots and so on, in images of people. Typical approaches for PPE detection based on deep learning are (i) to train an object detector for items such as those listed above or (ii) to train a person detector and a classifier that takes the bounding boxes predicted by the detector and discriminates between people wearing and people not wearing the corresponding PPE items. We propose a novel and accurate approach that uses three components: a person detector, a body pose estimator and a classifier. Our novelty consists in using the pose estimator only at training time, to improve the prediction performance of the classifier. We modify the neural architecture of the classifier by adding a spatial attention mechanism, which is trained using supervision signal from the pose estimator. In this way, the classifier learns to focus on PPE items, using knowledge from the pose estimator with almost no computational overhead during inference.

[1]  Jie Li,et al.  Safety helmet wearing detection based on image processing and machine learning , 2017, 2017 Ninth International Conference on Advanced Computational Intelligence (ICACI).

[2]  Luc Van Gool,et al.  The Pascal Visual Object Classes Challenge: A Retrospective , 2014, International Journal of Computer Vision.

[3]  Oishila Bandyopadhyay,et al.  Automated Helmet Detection for Multiple Motorcycle Riders using CNN , 2019, 2019 IEEE Conference on Information and Communication Technology.

[4]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[5]  Lisheng Xu,et al.  Hardhat-Wearing Detection Based on a Lightweight Convolutional Neural Network with Multi-Scale Features and a Top-Down Module , 2020, Sensors.

[6]  Romuere Rodrigues Veloso e Silva,et al.  Helmet Detection on Motorcyclists Using Image Descriptors and Classifiers , 2014, 2014 27th SIBGRAPI Conference on Graphics, Patterns and Images.

[7]  Achim J. Lilienthal,et al.  A Customized Vision System for Tracking Humans Wearing Reflective Safety Clothing from Industrial Vehicles and Machinery , 2014, Sensors.

[8]  Amir H. Behzadan,et al.  Deep learning for site safety: Real-time detection of personal protective equipment , 2020 .

[9]  Jixiu Wu,et al.  Automatic detection of hardhats worn by construction personnel: A deep learning approach and benchmark dataset , 2019, Automation in Construction.

[10]  Xiaochun Luo,et al.  Detecting non-hardhat-use by a deep learning method from far-field surveillance videos , 2018 .

[11]  Ali Farhadi,et al.  YOLOv3: An Incremental Improvement , 2018, ArXiv.

[12]  Evangelos A. Yfantis,et al.  Hard-Hat Detection for Construction Safety Visualization , 2015 .

[13]  Marc Miska,et al.  A real-time computer visionsystem for workers' PPE and posture detection in actual construction site environment , 2019 .

[14]  C. Krishna Mohan,et al.  Automatic detection of bike-riders without helmet using surveillance videos in real-time , 2016, 2016 International Joint Conference on Neural Networks (IJCNN).

[15]  Romuere Rôdrigues Veloso e Silva,et al.  Helmet Detection on Motorcyclists Using Image Descriptors and Classifiers , 2014, SIBGRAPI.

[16]  Tiago M. Fernández-Caramés,et al.  Real-time personal protective equipment monitoring system , 2012, Comput. Commun..

[17]  Puran Singh,et al.  Automatic Helmet Detection in Real-Time and Surveillance Video , 2020 .

[18]  Yaser Sheikh,et al.  OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[19]  Mark Sandler,et al.  MobileNetV2: Inverted Residuals and Linear Bottlenecks , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[20]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[21]  Li Zewen,et al.  A convolutional neural network based approach towards real-time hard hat detection , 2018, PIC 2018.

[22]  Mongkol Ekpanyapong,et al.  Low Cost, High Performance Automatic Motorcycle Helmet Violation Detection , 2020, 2020 IEEE Winter Conference on Applications of Computer Vision (WACV).

[23]  Yuechao He,et al.  A convolutional neural network based approach towards real-time hard hat detection , 2018, 2018 IEEE International Conference on Progress in Informatics and Computing (PIC).

[24]  Endah Suryawati Ningrum,et al.  Computer Vision System Based for Personal Protective Equipment Detection, by Using Convolutional Neural Network , 2019, 2019 International Electronics Symposium (IES).

[25]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[26]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[27]  Bastian Leibe,et al.  Multi-band Hough Forests for detecting humans with Reflective Safety Clothing from mobile machinery , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[28]  Pietro Perona,et al.  Microsoft COCO: Common Objects in Context , 2014, ECCV.

[29]  Ali Farhadi,et al.  You Only Look Once: Unified, Real-Time Object Detection , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Hiam Khoury,et al.  Automated Hardhat Detection for Construction Safety Applications , 2017 .

[31]  Hiam Khoury,et al.  Vision-Based Framework for Intelligent Monitoring of Hardhat Wearing on Construction Sites , 2019, J. Comput. Civ. Eng..

[32]  Hao Wu,et al.  An intelligent vision-based approach for helmet identification for work safety , 2018, Comput. Ind..

[33]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[34]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.