Crowded Scene Analysis: A Survey

Automated scene analysis has been a topic of great interest in computer vision and cognitive science. Recently, with the growth of crowd phenomena in the real world, crowded scene analysis has attracted much attention. However, the visual occlusions and ambiguities in crowded scenes, as well as the complex behaviors and scene semantics, make the analysis a challenging task. In the past few years, an increasing number of works on the crowded scene analysis have been reported, which covered different aspects including crowd motion pattern learning, crowd behavior and activity analyses, and anomaly detection in crowds. This paper surveys the state-of-the-art techniques on this topic. We first provide the background knowledge and the available features related to crowded scenes. Then, existing models, popular algorithms, evaluation protocols, and system performance are provided corresponding to different aspects of the crowded scene analysis. We also outline the available datasets for performance evaluation. Finally, some research problems and promising future directions are presented with discussions.

[1]  R. Brambilla,et al.  For pedestrians only : planning, design, and management of traffic-free zones , 1977 .

[2]  Daniel S. Hirschberg,et al.  Algorithms for the Longest Common Subsequence Problem , 1977, JACM.

[3]  C. Tomasi Detection and Tracking of Point Features , 1991 .

[4]  J. Sime Crowd psychology and engineering , 1995 .

[5]  P. Molnár Social Force Model for Pedestrian Dynamics Typeset Using Revt E X 1 , 1995 .

[6]  Helbing,et al.  Social force model for pedestrian dynamics. , 1995, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.

[7]  Soraia Raupp Musse,et al.  A Model of Human Crowd Behavior : Group Inter-Relationship and Collision Detection Analysis , 1997, Computer Animation and Simulation.

[8]  Jarke J. van Wijk,et al.  Image based flow visualization , 2002, ACM Trans. Graph..

[9]  Dimitrios Gunopulos,et al.  Discovering similar multidimensional trajectories , 2002, Proceedings 18th International Conference on Data Engineering.

[10]  Roger L. Hughes,et al.  A continuum theory for the flow of pedestrians , 2002 .

[11]  Tieniu Tan,et al.  A survey on visual surveillance of object motion and behaviors , 2004, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[12]  Richard M. Leggett,et al.  Real-Time Crowd Simulation: A Review , 2004 .

[13]  Thomas Brox,et al.  High Accuracy Optical Flow Estimation Based on a Theory for Warping , 2004, ECCV.

[14]  Robert Berggren Simulating Crowd Behaviour in Computer Games , 2005 .

[15]  Ernesto Andrade,et al.  Simulation of Crowd Problems for Computer Vision , 2005 .

[16]  J. Marsden,et al.  Definition and properties of Lagrangian coherent structures from finite-time Lyapunov exponents in two-dimensional aperiodic flows , 2005 .

[17]  Dmitry Chetverikov,et al.  A Brief Survey of Dynamic Texture Description and Recognition , 2005, CORES.

[18]  Ann B. Lee,et al.  Diffusion maps and coarse-graining: a unified framework for dimensionality reduction, graph partitioning, and data set parameterization , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[19]  Adrien Treuille,et al.  Continuum crowds , 2006, SIGGRAPH 2006.

[20]  M. Schreckenberg,et al.  Experimental study of pedestrian flow through a bottleneck , 2006, physics/0610077.

[21]  Tieniu Tan,et al.  A system for learning statistical motion patterns , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[22]  W. Eric L. Grimson,et al.  Unsupervised Activity Perception by Hierarchical Bayesian Models , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[23]  Liqing Zhang,et al.  Saliency Detection: A Spectral Residual Approach , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[24]  Mubarak Shah,et al.  A Lagrangian Particle Dynamics Approach for Crowd Flow Segmentation and Stability Analysis , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[25]  M. Shah,et al.  Taming crowded visual scenes , 2008 .

[26]  Dmitry B. Goldgof,et al.  How effective is human video surveillance performance? , 2008, 2008 19th International Conference on Pattern Recognition.

[27]  Mubarak Shah,et al.  Learning motion patterns in crowded scenes using motion flow field , 2008, 2008 19th International Conference on Pattern Recognition.

[28]  Anil M. Cheriyadat,et al.  Detecting Dominant Motions in Dense Crowds , 2008, IEEE Journal of Selected Topics in Signal Processing.

[29]  Mohan M. Trivedi,et al.  A Survey of Vision-Based Trajectory Learning and Analysis for Surveillance , 2008, IEEE Transactions on Circuits and Systems for Video Technology.

[30]  Mubarak Shah,et al.  Detecting global motion patterns in complex videos , 2008, 2008 19th International Conference on Pattern Recognition.

[31]  Nuno Vasconcelos,et al.  Modeling, Clustering, and Segmenting Video with Mixtures of Dynamic Textures , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[32]  Dirk Helbing,et al.  From Crowd Dynamics to Crowd Safety: a Video-Based Analysis , 2008, Adv. Complex Syst..

[33]  Sergio A. Velastin,et al.  Crowd analysis: a survey , 2008, Machine Vision and Applications.

[34]  Mubarak Shah,et al.  Floor Fields for Tracking in High Density Crowd Scenes , 2008, ECCV.

[35]  W. Eric L. Grimson,et al.  Unsupervised Activity Perception in Crowded and Complicated Scenes Using Hierarchical Bayesian Models , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[36]  Takeo Kanade,et al.  Tracking in unstructured crowded scenes , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[37]  L. Kratz,et al.  Anomaly detection in extremely crowded scenes using spatio-temporal motion pattern models , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[38]  Ce Liu,et al.  Exploring new representations and applications for motion analysis , 2009 .

[39]  Mubarak Shah,et al.  Video Scene Understanding Using Multi-scale Analysis , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[40]  M. Shah,et al.  Abnormal crowd behavior detection using social force model , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[41]  Nicolas Courty,et al.  Crowd Flow Characterization with Optimal Control Theory , 2009, ACCV.

[42]  Xuan Song,et al.  An online approach: Learning-Semantic-Scene-by-Tracking and Tracking-by-Learning-Semantic-Scene , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[43]  John R. Hershey,et al.  Single-Channel Multitalker Speech Recognition , 2010, IEEE Signal Processing Magazine.

[44]  Mubarak Shah,et al.  A Streakline Representation of Flow in Crowded Scenes , 2010, ECCV.

[45]  Mubarak Shah,et al.  Scene understanding by statistical modeling of motion patterns , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[46]  Nuno Vasconcelos,et al.  Anomaly detection in crowded scenes , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[47]  Mubarak Shah,et al.  Chaotic invariants of Lagrangian particle trajectories for anomaly detection in crowded scenes , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[48]  Soraia Raupp Musse,et al.  Crowd Analysis Using Computer Vision Techniques , 2010, IEEE Signal Processing Magazine.

[49]  Ramakant Nevatia,et al.  Multi-target tracking by on-line learned discriminative appearance models , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[50]  Chabane Djeraba,et al.  Motion Pattern Extraction and Event Detection for Automatic Visual Surveillance , 2011, EURASIP J. Image Video Process..

[51]  Xinyu Wu,et al.  Abnormal crowd behavior detection based on the energy model , 2011, 2011 IEEE International Conference on Information and Automation.

[52]  Alessio Del Bue,et al.  Optimizing interaction force for global anomaly detection in crowded scenes , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[53]  Zhongke Shi,et al.  Understanding dynamic scenes by hierarchical motion pattern mining , 2011, 2011 IEEE International Conference on Multimedia and Expo.

[54]  Bo Wang,et al.  Abnormal crowd behavior detection using high-frequency and spatio-temporal features , 2011, Machine Vision and Applications.

[55]  Ying Liu,et al.  Crowd density estimation based on image potential energy model , 2011, 2011 IEEE International Conference on Robotics and Biomimetics.

[56]  Alessio Del Bue,et al.  Abnormal Crowd Behavior Detection by Social Force Optimization , 2011, HBU.

[57]  Bo Wang,et al.  Abnormal crowd behavior detection using size-adapted spatio-temporal features , 2011 .

[58]  Jing Zhao,et al.  Crowd instability analysis using velocity-field based social force model , 2011, 2011 Visual Communications and Image Processing (VCIP).

[59]  Gérard G. Medioni,et al.  Robust unsupervised motion pattern inference from video and applications , 2011, 2011 International Conference on Computer Vision.

[60]  Tao Xiang,et al.  Identifying Rare and Subtle Behaviors: A Weakly Supervised Joint Topic Model , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[61]  Shaogang Gong,et al.  Visual Analysis of Behaviour , 2011 .

[62]  Ivan Laptev,et al.  Data-driven crowd analysis in videos , 2011, ICCV.

[63]  Shaogang Gong,et al.  Video Behaviour Mining Using a Dynamic Topic Model , 2011, International Journal of Computer Vision.

[64]  Siti Zaiton Mohd Hashim,et al.  Crowd Analysis and Its Applications , 2011, ICSECS.

[65]  Xiaogang Wang,et al.  Random field topic model for semantic region analysis in crowded scenes from tracklets , 2011, CVPR 2011.

[66]  Sridha Sridharan,et al.  Dynamic texture reconstruction from sparse codes for unusual event detection in crowded scenes , 2011, J-MRE '11.

[67]  Zhijing Liu,et al.  Motion pattern analysis in crowded scenes by using density based clustering , 2012, 2012 9th International Conference on Fuzzy Systems and Knowledge Discovery.

[68]  Xiaogang Wang,et al.  Coherent Filtering: Detecting Coherent Motions from Crowd Clutters , 2012, ECCV.

[69]  Kejun Wang,et al.  Video-Based Abnormal Human Behavior Recognition—A Review , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[70]  Shaogang Gong,et al.  Salient motion detection in crowded scenes , 2012, 2012 5th International Symposium on Communications, Control and Signal Processing.

[71]  Hanqing Lu,et al.  Learning Semantic Motion Patterns for Dynamic Scenes by Improved Sparse Topical Coding , 2012, 2012 IEEE International Conference on Multimedia and Expo.

[72]  Mubarak Shah,et al.  Identifying Behaviors in Crowd Scenes Using Stability Analysis for Dynamical Systems , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[73]  Shaogang Gong,et al.  Feature Mining for Localised Crowd Counting , 2012, BMVC.

[74]  Christian Bauckhage,et al.  Loveparade 2010: Automatic video analysis of a crowd disaster , 2012, Comput. Vis. Image Underst..

[75]  Xiaogang Wang,et al.  Understanding collective crowd behaviors: Learning a Mixture model of Dynamic pedestrian-Agents , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[76]  Yann LeCun,et al.  Road Scene Segmentation from a Single Image , 2012, ECCV.

[77]  Gérard G. Medioni,et al.  Tracking Using Motion Patterns for Very Crowded Scenes , 2012, ECCV.

[78]  Shuang Wu,et al.  Abnormal crowd behavior detection based on local pressure model , 2012, Proceedings of The 2012 Asia Pacific Signal and Information Processing Association Annual Summit and Conference.

[79]  Siti Zaiton Mohd Hashim,et al.  DETECTION OF ABNORMAL BEHAVIORS IN CROWD SCENE: A REVIEW , 2012, SOCO 2012.

[80]  Hau-San Wong,et al.  Crowd Motion Partitioning in a Scattered Motion Field , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[81]  Brett J. Borghetti,et al.  A Review of Anomaly Detection in Automated Surveillance , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[82]  Guohui Li,et al.  Abnormal crowd behavior detection using behavior entropy model , 2012, 2012 International Conference on Wavelet Analysis and Pattern Recognition.

[83]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[84]  Ko Nishino,et al.  Tracking Pedestrians Using Local Spatio-Temporal Motion Patterns in Extremely Crowded Scenes , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[85]  Duc Phu Chau,et al.  Multi-target tracking by discriminative analysis on Riemannian manifold , 2012, 2012 19th IEEE International Conference on Image Processing.

[86]  Paolo Remagnino,et al.  Laplacian Eigenmap With Temporal Constraints for Local Abnormality Detection in Crowded Scenes , 2013, IEEE Transactions on Cybernetics.

[87]  Mario Vento,et al.  Counting moving persons in crowded scenes , 2013, Machine Vision and Applications.

[88]  Xiaogang Wang,et al.  Multi-stage Contextual Deep Learning for Pedestrian Detection , 2013, 2013 IEEE International Conference on Computer Vision.

[89]  Meng Wang,et al.  Detecting Group Activities With Multi-Camera Context , 2013, IEEE Transactions on Circuits and Systems for Video Technology.

[90]  Tiejun Huang,et al.  Selective Eigenbackground for Background Modeling and Subtraction in Crowded Scenes , 2013, IEEE Transactions on Circuits and Systems for Video Technology.

[91]  D. Forsyth,et al.  Video Event Detection: From Subvolume Localization To Spatio-Temporal Path Search. , 2013, IEEE transactions on pattern analysis and machine intelligence.

[92]  Pierre-Marc Jodoin,et al.  Meta-tracking for video scene understanding , 2013, 2013 10th IEEE International Conference on Advanced Video and Signal Based Surveillance.

[93]  Duan-Yu Chen,et al.  Visual-Based Human Crowds Behavior Analysis Based on Graph Modeling and Matching , 2013, IEEE Sensors Journal.

[94]  Hua Yang,et al.  The Large-Scale Crowd Behavior Perception Based on Spatio-Temporal Viscous Fluid Field , 2013, IEEE Transactions on Information Forensics and Security.

[95]  Yandong Tang,et al.  Video Anomaly Search in Crowded Scenes via Spatio-Temporal Motion Context , 2013, IEEE Transactions on Information Forensics and Security.

[96]  Zhe Wu,et al.  Motion pattern analysis in crowded scenes based on hybrid generative-discriminative feature maps , 2013, 2013 IEEE International Conference on Image Processing.

[97]  Martin D. Levine,et al.  An on-line, real-time learning method for detecting anomalies in videos using spatio-temporal compositions , 2013, Comput. Vis. Image Underst..

[98]  Dimitrios Makris,et al.  Tracklet Reidentification in Crowded Scenes Using Bag of Spatio-temporal Histograms of Oriented Gradients , 2013, MCPR.

[99]  Antoni B. Chan,et al.  Crossing the Line: Crowd Counting by Integer Programming with Local Features , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[100]  How-Lung Eng,et al.  A Literature Review on Video Analytics of Crowded Scenes , 2013, Intelligent Multimedia Surveillance.

[101]  Junsong Yuan,et al.  Abnormal event detection in crowded scenes using sparse representation , 2013, Pattern Recognit..

[102]  H. Zha,et al.  A fully online and unsupervised system for large and high-density area surveillance: Tracking, semantic scene learning and abnormality detection , 2013, TIST.

[103]  James M. Rehg,et al.  Video-Based Crowd Synthesis , 2013, IEEE Transactions on Visualization and Computer Graphics.

[104]  Liu Yuncai,et al.  Analyzing motion patterns in crowded scenes via automatic tracklets clustering , 2013, China Communications.

[105]  David A. Forsyth,et al.  Video Event Detection: From Subvolume Localization to Spatiotemporal Path Search , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[106]  Zhiwen Yu,et al.  A Bayesian Model for Crowd Escape Behavior Detection , 2014, IEEE Transactions on Circuits and Systems for Video Technology.

[107]  Nuno Vasconcelos,et al.  Anomaly Detection and Localization in Crowded Scenes , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[108]  Xiaofei Wang,et al.  A high accuracy flow segmentation method in crowded scenes based on streakline , 2014 .

[109]  Sophie Papst,et al.  Computational Methods For Fluid Dynamics , 2016 .