Dynamic prioritization of surveillance video data in real-time automated detection systems

Abstract Automated object detection systems are a key component of modern surveillance applications. These systems rely on computationally expensive computer vision algorithms that perform object detection on visual data recorded by surveillance cameras. Due to the security and safety implications of these systems, this visual data st be processed accurately and in real-time. However, many of the frames that are created by the surveillance cameras may be of low importance, providing little or no useful information to the object detection system. Sub-sampling surveillance data by prioritizing important camera frames can greatly reduce unnecessary computation. Consequently, several works have explored dynamic visual data sub-sampling using various modalities of information (ie. spatial or temporal information) for prioritization. Few works, however, have combined and evaluated different modalities of information together for real-time prioritization of visual surveillance data. This work evaluates several individual and combined prioritization metrics derived from different modalities of information for use with a modern deep learning-based object detection algorithm. Both processing time and object detection rate are measured and used to rank the prioritization metrics. A novel approach that uses the historical detection confidences created by the object detection algorithm was demonstrated to be the best standalone prioritization metric. Additionally, a novel ensemble method that uses a KNN regressor to combine the best of the previously evaluated metrics to create a dynamic prioritization method is presented. This ensemble approach is shown to increase the object detection rate by up to 60% as compared to a static sub-sampling baseline as demonstrated using three publicly available datasets. The increased object detection rate was achieved while meeting the real-time constraints of the automated object detection system.