Crowd behavior representation: an attribute-based approach

In crowd behavior studies, a model of crowd behavior needs to be trained using the information extracted from video sequences. Most of the previous methods are based on low-level visual features because there are only crowd behavior labels available as ground-truth information in crowd datasets. However, there is a huge semantic gap between low-level motion/appearance features and high-level concept of crowd behaviors. In this paper, we tackle the problem by introducing an attribute-based scheme. While similar strategies have been employed for action and object recognition, to the best of our knowledge, for the first time it is shown that the crowd emotions can be used as attributes for crowd behavior understanding. We explore the idea of training a set of emotion-based classifiers, which can subsequently be used to indicate the crowd motion. In this scheme, we collect a large dataset of video clips and provide them with both annotations of “crowd behaviors” and “crowd emotions”. We test the proposed emotion based crowd representation methods on our dataset. The obtained promising results demonstrate that the crowd emotions enable the construction of more descriptive models for crowd behaviors. We aim at publishing the dataset with the article, to be used as a benchmark for the communities.

[1]  Philip R. Cohen,et al.  Emotional influences in memory and thinking: Data and theory , 1982 .

[2]  Ivan Laptev,et al.  Data-driven crowd analysis in videos , 2011, ICCV.

[3]  J. Ferryman,et al.  An overview of the PETS 2009 challenge , 2009 .

[4]  Mubarak Shah,et al.  Abnormal crowd behavior detection using social force model , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[5]  D. Cliff.  From animals to animats , 1994, Nature.

[6]  Dirk Helbing,et al.  Recognition of crowd behavior from mobile sensors with pattern analysis and graph clustering methods , 2011, Networks Heterog. Media.

[7]  Christian Bauckhage,et al.  Analyzing pedestrian behavior in crowds for automatic detection of congestions , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[8]  Silvio Savarese,et al.  Recognizing human actions by attributes , 2011, CVPR 2011.

[9]  Hua Yang,et al.  The Large-Scale Crowd Behavior Perception Based on Spatio-Temporal Viscous Fluid Field , 2013, IEEE Transactions on Information Forensics and Security.

[10]  François Brémond,et al.  Crowd Behavior Recognition for Video Surveillance , 2008, ACIVS.

[11]  Ali Farhadi,et al.  Attribute-centric recognition for cross-category generalization , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[12]  P. Ekman.  An argument for basic emotions , 1992 .

[13]  Kjell Elenius,et al.  Emotion Recognition , 2009, Computers in the Human Interaction Loop.

[14]  Alessio Del Bue,et al.  Optimizing interaction force for global anomaly detection in crowded scenes , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[15]  Qingming Huang,et al.  Abnormal crowd behavior detection based on social attribute-aware force model , 2012, 2012 19th IEEE International Conference on Image Processing.

[16]  Ali Farhadi,et al.  Describing objects by their attributes , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[17]  Bo Wang,et al.  Abnormal crowd behavior detection using high-frequency and spatio-temporal features , 2011, Machine Vision and Applications.

[18]  Yangsheng Xu,et al.  Abnormal Behavior Detection by Multi-SVM-Based Bayesian Network , 2007, 2007 International Conference on Information Acquisition.

[19]  Nuno Vasconcelos,et al.  Anomaly detection in crowded scenes , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[20]  Cordelia Schmid,et al.  Learning realistic human actions from movies , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[21]  Mubarak Shah,et al.  Identifying Behaviors in Crowd Scenes Using Stability Analysis for Dynamical Systems , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[22]  Nuno Vasconcelos,et al.  Anomaly Detection and Localization in Crowded Scenes , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[23]  Christoph H. Lampert,et al.  Learning to detect unseen object classes by between-class attribute transfer , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[24]  W. Eric L. Grimson,et al.  Unsupervised Activity Perception in Crowded and Complicated Scenes Using Hierarchical Bayesian Models , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[25]  Ko Nishino,et al.  Tracking with local spatio-temporal motion patterns in extremely crowded scenes , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[26]  Arthur C. Graesser,et al.  Automatic detection of learner’s affect from conversational cues , 2008, User Modeling and User-Adapted Interaction.

[27]  George N. Votsis,et al.  Emotion recognition in human-computer interaction , 2001, IEEE Signal Process. Mag..

[28]  Ramin Mehran,et al.  Abnormal crowd behavior detection using social force model , 2009, CVPR.

[29]  Tal Hassner,et al.  Violent flows: Real-time detection of violent crowd behavior , 2012, 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[30]  David A. McAllester,et al.  A discriminatively trained, multiscale, deformable part model , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[31]  A. Goldman,et al.  Simulationist models of face-based emotion recognition , 2005, Cognition.

[32]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[33]  P. Ekman.  Expression and the Nature of Emotion , 1984 .

[34]  Cordelia Schmid,et al.  Action recognition by dense trajectories , 2011, CVPR 2011.

[35]  Yang Wang,et al.  Max-margin hidden conditional random fields for human action recognition , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[36]  Matthias Rauterberg,et al.  Crowd Emotion Detection Using Dynamic Probabilistic Models , 2014, SAB.

[37]  L. Kratz,et al.  Anomaly detection in extremely crowded scenes using spatio-temporal motion pattern models , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[38]  Rachel McDonnell,et al.  Perceiving emotion in crowds: the role of dynamic body postures on the perception of emotion in crowded scenes , 2010, Experimental Brain Research.

[39]  Rosalind W. Picard,et al.  Automated Posture Analysis for Detecting Learner's Interest Level , 2003, 2003 Conference on Computer Vision and Pattern Recognition Workshop.

[40]  Mark C. Coulson.  Attributing Emotion to Static Body Postures: Recognition Accuracy, Confusions, and Viewpoint Dependence , 2004 .

[41]  Yoshua Bengio,et al.  Zero-data Learning of New Tasks , 2008, AAAI.

[42]  Matthias Rauterberg,et al.  Perception of emotions from crowd dynamics , 2015, 2015 IEEE International Conference on Digital Signal Processing (DSP).

[43]  Christian Bauckhage,et al.  Loveparade 2010: Automatic video analysis of a crowd disaster , 2012, Comput. Vis. Image Underst..

[44]  Cordelia Schmid,et al.  Human Detection Using Oriented Histograms of Flow and Appearance , 2006, ECCV.

[45]  Björn W. Schuller,et al.  Hidden Markov model-based speech emotion recognition , 2003, 2003 International Conference on Multimedia and Expo. ICME '03. Proceedings (Cat. No.03TH8698).