D-STC: Deep learning with spatio-temporal constraints for train drivers detection from videos

Abstract Video-based train driver operation monitoring is one of the emerging requirements for train safety management and driving operation regularization. Recent years, deep learning methods such as Faster R-CNN have achieved excellent detection performance on images. However, they are not specially designed for object detection from videos, especially for those train drivers who often perform tiny moving in the monitoring video. Spatial and temporal information of videos are not fully explored together to solve this problem. In this paper, a new framework D-STC is proposed to handles the complex situations in train cab and detect train drivers from videos in a more reliable way. The proposed framework first utilizes fine tuning Faster R-CNN framework to detect the train drivers as the initial detection results. Then, the initial detection results of each frame is processed further to suppress false detection results by using the customized spatial constraints. Finally, an optimal threshold adjustment mechanism is presented to improve detection accuracy for the whole video sequence. The D-STC framework improves the accuracy of train driver detection and fully guarantees the detection speed for videos. Experimental results demonstrate the effectiveness of the proposed framework.