A fully online and unsupervised system for large and high-density area surveillance: Tracking, semantic scene learning and abnormality detection

For reasons of public security, an intelligent surveillance system that can cover a large, crowded public area has become an urgent need. In this article, we propose a novel laser-based system that can simultaneously perform tracking, semantic scene learning, and abnormality detection in a fully online and unsupervised way. Furthermore, these three tasks cooperate with each other in one framework to improve their respective performances. The proposed system has the following key advantages over previous ones: (1) It can cover quite a large area (more than 60×35m), and simultaneously perform robust tracking, semantic scene learning, and abnormality detection in a high-density situation. (2) The overall system can vary with time, incrementally learn the structure of the scene, and perform fully online abnormal activity detection and tracking. This feature makes our system suitable for real-time applications. (3) The surveillance tasks are carried out in a fully unsupervised manner, so that there is no need for manual labeling and the construction of huge training datasets. We successfully apply the proposed system to the JR subway station in Tokyo, and demonstrate that it can cover an area of 60×35m, robustly track more than 150 targets at the same time, and simultaneously perform online semantic scene learning and abnormality detection with no human intervention.

[1]  Ryosuke Shibasaki,et al.  Fusion of Detection and Matching Based Approaches for Laser Based Multiple People Tracking , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[2]  Dan Schonfeld,et al.  Real-time interactively distributed multi-object tracking using a magnetic-inertia potential model , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[3]  Ali Shahrokni,et al.  Video Activity Extraction and Reporting with Incremental Unsupervised Learning , 2010, 2010 7th IEEE International Conference on Advanced Video and Signal Based Surveillance.

[4]  Tieniu Tan,et al.  Similarity based vehicle trajectory clustering and anomaly detection , 2005, IEEE International Conference on Image Processing 2005.

[5]  Stefan Carlsson,et al.  Tracking and Labelling of Interacting Multiple Targets , 2006, ECCV.

[6]  Wolfram Burgard,et al.  People Tracking with Mobile Robots Using Sample-Based Joint Probabilistic Data Association Filters , 2003, Int. J. Robotics Res..

[7]  Tim J. Ellis,et al.  Automatic learning of an activity-based semantic scene model , 2003, Proceedings of the IEEE Conference on Advanced Video and Signal Based Surveillance, 2003..

[8]  Nuno Vasconcelos,et al.  Anomaly detection in crowded scenes , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[9]  W. Eric L. Grimson,et al.  Trajectory analysis and semantic region modeling using a nonparametric Bayesian model , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[10]  Xuan Song,et al.  An online approach: Learning-Semantic-Scene-by-Tracking and Tracking-by-Learning-Semantic-Scene , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[11]  Mubarak Shah,et al.  Abnormal crowd behavior detection using social force model , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[12]  René Vidal,et al.  Online Clustering of Moving Hyperplanes , 2006, NIPS.

[13]  Ramakant Nevatia,et al.  Tracking multiple humans in complex situations , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  Margrit Betke,et al.  Tracking Large Variable Numbers of Objects in Clutter , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[15]  Ming-Hsuan Yang,et al.  Visual tracking with online Multiple Instance Learning , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[16]  O. Sebakhy,et al.  State regulation in linear discrete-time systems in minimum time , 1979 .

[17]  François Brémond,et al.  Trajectory Based Activity Discovery , 2010, 2010 7th IEEE International Conference on Advanced Video and Signal Based Surveillance.

[18]  Biswajit Bose,et al.  Multi-class object tracking algorithm that handles fragmentation and grouping , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[19]  Mubarak Shah,et al.  Probabilistic Modeling of Scene Dynamics for Applications in Visual Surveillance , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[20]  Joe W. Harris,et al.  Algebraic Geometry: A First Course , 1995 .

[21]  Shaogang Gong,et al.  International Journal of Computer Vision (The original publication is available at www.springerlink.com) Time-Delayed Correlation Analysis for Multi-Camera Activity Understanding , 2009 .

[22]  Shai Avidan,et al.  Ensemble Tracking , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[23]  Simon J. Godsill,et al.  On sequential Monte Carlo sampling methods for Bayesian filtering , 2000, Stat. Comput..

[24]  Imran N. Junejo,et al.  Multi feature path modeling for video surveillance , 2004, ICPR 2004.

[25]  Shai Avidan.  Ensemble Tracking , 2007, IEEE Trans. Pattern Anal. Mach. Intell..

[26]  W. Eric L. Grimson,et al.  Learning Semantic Scene Models by Trajectory Analysis , 2006, ECCV.

[27]  Gérard G. Medioni,et al.  Multiple Target Tracking Using Spatio-Temporal Markov Chain Monte Carlo Data Association , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[28]  Zhouyu Fu,et al.  Semantic-Based Surveillance Video Retrieval , 2007, IEEE Transactions on Image Processing.

[29]  Ramin Mehran,et al.  Abnormal crowd behavior detection using social force model , 2009, CVPR.

[30]  Mubarak Shah,et al.  Multi feature path modeling for video surveillance , 2004, Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004..

[31]  Y. Bar-Shalom.  Tracking and data association , 1988 .

[32]  Stefan Carlsson,et al.  Multi-Target Tracking - Linking Identities using Bayesian Network Inference , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[33]  Donald Reid.  An algorithm for tracking multiple targets , 1978 .

[34]  Xuan Song,et al.  Tracking interacting targets with laser scanner via on-line supervised learning , 2008, 2008 IEEE International Conference on Robotics and Automation.

[35]  James J. Little,et al.  A Linear Programming Approach for Multiple Object Tracking , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[36]  SongXuan,et al.  A fully online and unsupervised system for large and high-density area surveillance , 2013 .

[37]  Gregory D. Hager,et al.  Probabilistic data association methods in visual tracking of groups , 2004, CVPR 2004.

[38]  Shaogang Gong,et al.  Video Behavior Profiling for Anomaly Detection , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[39]  Mubarak Shah,et al.  Floor Fields for Tracking in High Density Crowd Scenes , 2008, ECCV.

[40]  Frank Dellaert,et al.  MCMC Data Association and Sparse Factorization Updating for Real Time Multitarget Tracking with Merged and Multiple Measurements , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[41]  W. Eric L. Grimson,et al.  Correspondence-Free Activity Analysis and Scene Modeling in Multiple Camera Views , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[42]  Mubarak Shah,et al.  Chaotic invariants of Lagrangian particle trajectories for anomaly detection in crowded scenes , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[43]  Tianzhu Zhang,et al.  Learning semantic scene models by object classification and trajectory clustering , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[44]  Xuan Song,et al.  Vision-Based Multiple Interacting Targets Tracking via On-Line Supervised Learning , 2008, ECCV.

[45]  S. Godsill,et al.  Monte Carlo filtering for multi target tracking and data association , 2005, IEEE Transactions on Aerospace and Electronic Systems.

[46]  Hassan Foroosh,et al.  Trajectory Rectification and Path Modeling for Video Surveillance , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[47]  Gregory D. Hager,et al.  Probabilistic Data Association Methods for Tracking Complex Visual Objects , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[48]  Ming Yang,et al.  Game-Theoretic Multiple Target Tracking , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[49]  Ehud Rivlin,et al.  Robust Real-Time Unusual Event Detection using Multiple Fixed-Location Monitors , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[50]  Xuan Song,et al.  A novel laser-based system: Fully online detection of abnormal activity via an unsupervised method , 2011, 2011 IEEE International Conference on Robotics and Automation.

[51]  VARUN CHANDOLA,et al.  Anomaly detection: A survey , 2009, CSUR.

[52]  Luc Van Gool,et al.  Coupled Detection and Trajectory Estimation for Multi-Object Tracking , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[53]  Roberto Manduchi,et al.  Hybrid joint-separable multibody tracking , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[54]  Hai Yang,et al.  ACM Transactions on Intelligent Systems and Technology - Special Section on Urban Computing , 2014 .

[55]  Ryosuke Shibasaki,et al.  A novel system for tracking pedestrians using multiple single-row laser-range scanners , 2005, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[56]  Dorin Comaniciu,et al.  Distribution Free Decomposition of Multivariate Data , 1998, Pattern Analysis & Applications.

[57]  Ming-Hsuan Yang,et al.  Incremental Learning for Robust Visual Tracking , 2008, International Journal of Computer Vision.