A comparative study between single and multi-frame anomaly detection and localization in recorded video streams

Abstract Video anomaly detection is usually studied by considering the spatial and temporal contexts. This paper focuses first on spatial context and shows that it can be a fast real-time solution. In the first part of this work there are two main contributions: employing a new deep network for reconstruction and introducing a new regularity scoring function. The new deep architecture is based on pyramid of input images and compared to UNet, the proposed architecture boosts AUC by 15% and the new regularity scoring function is based on SSIM. The second part employs a multiframe approach to distinguish temporal behavior anomalies. The second approach enhances the results by 7% compared to spatial anomaly detection. Comparing the two approaches, if computing power is limited and real time anomaly detection is looked for, single frame detection is preferred while multi frame analysis offers a much wider possibility of anomaly detection.

[1]  Nicu Sebe,et al.  Learning Deep Representations of Appearance and Motion for Anomalous Event Detection , 2015, BMVC.

[2]  Sanjay Chawla,et al.  Deep Learning for Anomaly Detection: A Survey , 2019, ArXiv.

[3]  Shenghua Gao,et al.  Future Frame Prediction for Anomaly Detection - A New Baseline , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[4]  Junsong Yuan,et al.  Sparse reconstruction cost for abnormal event detection , 2011, CVPR 2011.

[5]  M. Shah,et al.  Abnormal crowd behavior detection using social force model , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[6]  François Chollet,et al.  Xception: Deep Learning with Depthwise Separable Convolutions , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Yasin Yilmaz,et al.  Continual Learning for Anomaly Detection in Surveillance Videos , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[8]  Felix Naumann,et al.  Data fusion , 2009, CSUR.

[9]  Jonghyun Choi,et al.  Learning Temporal Regularity in Video Sequences , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Emmanuelle Gouillart,et al.  scikit-image: image processing in Python , 2014, PeerJ.

[11]  David P. Farrington,et al.  CCTV surveillance for crime prevention , 2019, Criminology & Public Policy.

[12]  Kamal Jamshidi,et al.  Anomaly detection and localization in crowded scenes using connected component analysis , 2017, Multimedia Tools and Applications.

[13]  Alireza Dehghani,et al.  Registration of fluorescein angiography and optical coherence tomography images of curved retina via scanning laser ophthalmoscopy photographs. , 2020, Biomedical optics express.

[14]  Bin Liu,et al.  Real-time Anomaly Detection with HMOF Feature , 2018, ICVIP.

[15]  Marc Litaudon,et al.  Collected mass spectrometry data on monoterpene indole alkaloids from natural product chemistry research , 2019, Scientific Data.

[16]  Chengdong Wu,et al.  An Efficient Anomaly Detection System for Crowded Scenes Using Variational Autoencoders , 2019 .

[17]  Long Ang Lim,et al.  Foreground segmentation using convolutional neural networks for multiscale feature encoding , 2018, Pattern Recognit. Lett..

[18]  Tomoyasu Horikawa,et al.  Characterization of deep neural network features by decodability from human brain activity , 2019, Scientific Data.

[19]  Mahmood Fathy,et al.  Fast and accurate detection and localization of abnormal behavior in crowded scenes , 2017, Machine Vision and Applications.

[20]  Kaiming He,et al.  A Multigrid Method for Efficiently Training Video Models , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Randy C. Paffenroth,et al.  Anomaly Detection with Robust Deep Autoencoders , 2017, KDD.

[22]  Christopher Leckie,et al.  High-dimensional and large-scale anomaly detection using a linear one-class SVM with deep learning , 2016, Pattern Recognit..

[23]  K. Grauman,et al.  Observe locally, infer globally: A space-time MRF for detecting abnormal activities with incremental updates , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[24]  Bhiksha Raj,et al.  On the Origin of Deep Learning , 2017, ArXiv.

[25]  Pascal Vincent,et al.  Representation Learning: A Review and New Perspectives , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[26]  VARUN CHANDOLA,et al.  Anomaly detection: A survey , 2009, CSUR.

[27]  Edward F. Chang,et al.  Speech synthesis from neural decoding of spoken sentences , 2019, Nature.

[28]  Jianfei Yang,et al.  Suppressing Uncertainties for Large-Scale Facial Expression Recognition , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  Kishanprasad G. Gunale,et al.  Deep Learning with a Spatiotemporal Descriptor of Appearance and Motion Estimation for Video Anomaly Detection , 2018, J. Imaging.

[30]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[31]  Mahmood Fathy,et al.  Adversarially Learned One-Class Classifier for Novelty Detection , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[32]  B. Ravi Kiran,et al.  An overview of deep learning based methods for unsupervised and semi-supervised anomaly detection in videos , 2018, J. Imaging.

[33]  Nicu Sebe,et al.  Training Adversarial Discriminators for Cross-Channel Abnormal Event Detection in Crowds , 2017, 2019 IEEE Winter Conference on Applications of Computer Vision (WACV).

[34]  Cewu Lu,et al.  Abnormal Event Detection at 150 FPS in MATLAB , 2013, 2013 IEEE International Conference on Computer Vision.

[35]  Young Min Kim,et al.  RL-GAN-Net: A Reinforcement Learning Agent Controlled GAN Network for Real-Time Point Cloud Shape Completion , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[36]  Nuno Vasconcelos,et al.  Anomaly detection in crowded scenes , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[37]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[38]  Mahmood Fathy,et al.  Deep-Cascade: Cascading 3D Deep Neural Networks for Fast Anomaly Detection and Localization in Crowded Scenes , 2017, IEEE Transactions on Image Processing.

[39]  Chunhua Shen,et al.  Self-Trained Deep Ordinal Regression for End-to-End Video Anomaly Detection , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[40]  Nicu Sebe,et al.  Abnormal event detection in videos using generative adversarial nets , 2017, 2017 IEEE International Conference on Image Processing (ICIP).

[41]  Jean Meunier,et al.  Anomaly Detection in Video Sequence With Appearance-Motion Correspondence , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[42]  Nuno Vasconcelos,et al.  Anomaly Detection and Localization in Crowded Scenes , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[43]  Douglas M. Hawkins.  Identification of Outliers , 1980, Monographs on Applied Probability and Statistics.

[44]  Heitor Silvério Lopes,et al.  A study of deep convolutional auto-encoders for anomaly detection in videos , 2018, Pattern Recognit. Lett..

[45]  Mahmood Fathy,et al.  Real-time anomaly detection and localization in crowded scenes , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[46]  Hongbin Zha,et al.  Learning to Detect Anomalies in Surveillance Video , 2015, IEEE Signal Processing Letters.

[47]  Shenghua Gao,et al.  Remembering history with convolutional LSTM for anomaly detection , 2017, 2017 IEEE International Conference on Multimedia and Expo (ICME).

[48]  Toby P. Breckon,et al.  GANomaly: Semi-Supervised Anomaly Detection via Adversarial Training , 2018, ACCV.

[49]  Meng Wang,et al.  Multimodal Deep Autoencoder for Human Pose Recovery , 2015, IEEE Transactions on Image Processing.

[50]  Jun Yu,et al.  Multimodal Face-Pose Estimation With Multitask Manifold Deep Learning , 2019, IEEE Transactions on Industrial Informatics.