Advances and Trends in Real Time Visual Crowd Analysis

Real time crowd analysis represents an active area of research within the computer vision community in general and scene analysis in particular. Over the last 10 years, various methods for crowd management in real time scenario have received immense attention due to large scale applications in people counting, public events management, disaster management, safety monitoring an so on. Although many sophisticated algorithms have been developed to address the task; crowd management in real time conditions is still a challenging problem being completely solved, particularly in wild and unconstrained conditions. In the proposed paper, we present a detailed review of crowd analysis and management, focusing on state-of-the-art methods for both controlled and unconstrained conditions. The paper illustrates both the advantages and disadvantages of state-of-the-art methods. The methods presented comprise the seminal research works on crowd management, and monitoring and then culminating state-of-the-art methods of the newly introduced deep learning methods. Comparison of the previous methods is presented, with a detailed discussion of the direction for future research work. We believe this review article will contribute to various application domains and will also augment the knowledge of the crowd analysis within the research community.

[1]  Ying Jin,et al.  Pedestrian monitoring techniques for crowd-flow prediction , 2017 .

[2]  Ryuzo Okada,et al.  COUNT Forest: CO-Voting Uncertain Number of Targets Using Random Forest for Crowd Density Estimation , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[3]  Shenghua Gao,et al.  Single-Image Crowd Counting via Multi-Column Convolutional Neural Network , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Shiv Surya,et al.  Switching Convolutional Neural Network for Crowd Counting , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Thomas C. Schmidt,et al.  Multi-Camera Crowd Monitoring: The SAFEST Approach , 2015 .

[6]  Khalil Khan,et al.  Intelligent Fusion of Deep Features for Improved Waste Classification , 2020, IEEE Access.

[7]  Yuhong Li,et al.  CSRNet: Dilated Convolutional Neural Networks for Understanding the Highly Congested Scenes , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[8]  Ivan Laptev,et al.  Density-aware person detection and tracking in crowds , 2011, ICCV.

[9]  Ali Al-Sheary,et al.  Crowd Monitoring System Using Unmanned Aerial Vehicle (UAV) , 2017 .

[10]  Jawad Ali Shah,et al.  Crowd Monitoring and Localization Using Deep Convolutional Neural Network: A Review , 2020, Applied Sciences.

[11]  Sridha Sridharan,et al.  Crowd Counting Using Multiple Local Features , 2009, 2009 Digital Image Computing: Techniques and Applications.

[12]  Takio Kurita,et al.  Mixture of Counting CNNs: Adaptive Integration of CNNs Specialized to Specific Appearance for Crowd Counting , 2017, ArXiv.

[13]  Khalil Khan,et al.  A Multi-Task Framework for Facial Attributes Classification through End-to-End Face Parsing and Deep Convolutional Neural Networks , 2020, Sensors.

[14]  Basim Zafar,et al.  GPU-based optimization of pilgrim simulation for hajj and umrah rituals , 2018 .

[15]  Ramakant Nevatia,et al.  Segmentation and Tracking of Multiple Humans in Crowded Environments , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[16]  Xiaogang Wang,et al.  Scene-Independent Group Profiling in Crowd , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[17]  Serge J. Belongie,et al.  Counting Crowded Moving Objects , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[18]  Ahmad Hoirul Basori,et al.  Development of 3D Tawaf Simulation for Hajj Training Application Using Virtual Environment , 2011, IVIC.

[19]  Shaogang Gong,et al.  Cumulative Attribute Space for Age and Crowd Density Estimation , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[20]  Jing Wang,et al.  Effective Crowd Anomaly Detection Through Spatio-temporal Texture Analysis , 2019, Int. J. Autom. Comput..

[21]  Nasir Ahmad,et al.  A framework for head pose estimation and face segmentation through conditional random fields , 2020, Signal Image Video Process..

[22]  Xiaogang Wang,et al.  Deeply learned attributes for crowded scene understanding , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Yu Wang,et al.  A Deeply-Recursive Convolutional Network For Crowd Counting , 2018, 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[24]  Jungong Han,et al.  ST-CNN: Spatial-Temporal Convolutional Neural Network for crowd counting in videos , 2019, Pattern Recognit. Lett..

[25]  Patrick J. Flynn,et al.  The effectiveness of face detection algorithms in unconstrained crowd scenes , 2014, IEEE Winter Conference on Applications of Computer Vision.

[26]  Antoni B. Chan,et al.  Small instance detection by integer programming on object density maps , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  Xiaogang Wang,et al.  Cross-scene crowd counting via deep convolutional neural networks , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Paul A. Viola,et al.  Robust Real-Time Face Detection , 2001, International Journal of Computer Vision.

[29]  Andrew Zisserman,et al.  Learning To Count Objects in Images , 2010, NIPS.

[30]  Wafa Alrajhi,et al.  Urban Analytics in Crowd Management in the Context of Hajj , 2016, HCI.

[31]  Roberto Cipolla,et al.  Unsupervised Bayesian Detection of Independent Motion in Crowds , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[32]  Nikos Paragios,et al.  A MRF-based approach for real-time subway monitoring , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[33]  Roberto Cipolla,et al.  SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[34]  Noel E. O'Connor,et al.  Fully Convolutional Crowd Counting on Highly Congested Scenes , 2016, VISIGRAPP.

[35]  Haroon Idrees,et al.  Multi-source Multi-scale Counting in Extremely Dense Crowd Images , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[36]  Kazunori Kotani,et al.  Saliency detection in human crowd images of different density levels using attention mechanism , 2020, Signal Process. Image Commun..

[37]  Weijia Li,et al.  Deep Learning Based Oil Palm Tree Detection and Counting for High-Resolution Remote Sensing Images , 2016, Remote. Sens..

[38]  Siti Zaiton Mohd Hashim,et al.  DETECTION OF ABNORMAL BEHAVIORS IN CROWD SCENE: A REVIEW , 2012, SOCO 2012.

[39]  Changyin Sun,et al.  Crowd Counting via Weighted VLAD on a Dense Attribute Feature Map , 2016, IEEE Transactions on Circuits and Systems for Video Technology.

[40]  Paul A. Viola,et al.  Detecting Pedestrians Using Patterns of Motion and Appearance , 2005, International Journal of Computer Vision.

[41]  Ashutosh Saxena,et al.  Counting the Number of People in Crowd as a Part of Automatic Crowd Monitoring: A Combined Approach , 2019 .

[42]  Yinfeng Fang,et al.  Detection of Salient Crowd Motion Based on Repulsive Force Network and Direction Entropy , 2019, Entropy.

[43]  Hassan M. Al-Ahmadi,et al.  Statistical analysis of the crowd dynamics in Al-Masjid Al-Nabawi in the city of Medina, Saudi Arabia , 2018 .

[44]  Cai Fudong,et al.  Abnormal Behavior Detection Based on Optical Flow Trajectory of Human Joint Points , 2019, 2019 Chinese Control And Decision Conference (CCDC).

[45]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[46]  Qi Wang,et al.  NWPU-Crowd: A Large-Scale Benchmark for Crowd Counting and Localization , 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[47]  Devis Tuia,et al.  Detecting Mammals in UAV Images: Best Practices to address a substantially Imbalanced Dataset with Deep Learning , 2018, Remote Sensing of Environment.

[48]  Yan Li,et al.  Multi-view crowd congestion monitoring system based on an ensemble of convolutional neural network classifiers , 2020, J. Intell. Transp. Syst..

[49]  Li Yang,et al.  Intelligent Crime Anomaly Detection in Smart Cities Using Deep Learning , 2018, 2018 IEEE 4th International Conference on Collaboration and Internet Computing (CIC).

[50]  Yi Wang,et al.  Fast visual object counting via example-based density estimation , 2016, 2016 IEEE International Conference on Image Processing (ICIP).

[51]  SogaKenichi,et al.  Pedestrian monitoring techniques for crowd-flow prediction , 2017 .

[52]  Tarik Taleb,et al.  UAV-Based IoT Platform: A Crowd Surveillance Use Case , 2017, IEEE Communications Magazine.

[53]  Quoc Cuong Pham,et al.  Crowd Behavior Analysis Using Local Mid-Level Visual Descriptors , 2017, IEEE Transactions on Circuits and Systems for Video Technology.

[54]  S. Tsaftaris,et al.  Learning to Count Leaves in Rosette Plants , 2015 .

[55]  Mohammed Talat Khouj,et al.  Disaster management in real time simulation using machine learning , 2011, 2011 24th Canadian Conference on Electrical and Computer Engineering(CCECE).

[56]  Qijun Zhao,et al.  Point in, Box Out: Beyond Counting Persons in Crowds , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[57]  Kate Saenko,et al.  Asymmetric and Category Invariant Feature Transformations for Domain Adaptation , 2014, International Journal of Computer Vision.

[58]  Xiaogang Wang,et al.  Learning Collective Crowd Behaviors with Dynamic Pedestrian-Agents , 2014, International Journal of Computer Vision.

[59]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[60]  Qi Tian,et al.  Recognizing human group action by layered model with multiple cues , 2014, Neurocomputing.

[61]  Yuan Yuan,et al.  Domain-adaptive Crowd Counting via Inter-domain Features Segregation and Gaussian-prior Reconstruction , 2019, ArXiv.

[62]  Mao Ye,et al.  Fast crowd density estimation with convolutional neural networks , 2015, Eng. Appl. Artif. Intell..

[63]  Haizhou Ai,et al.  End-to-end crowd counting via joint learning local and global count , 2016, 2016 IEEE International Conference on Image Processing (ICIP).

[64]  Brett Stevens,et al.  Scene perception guided crowd anomaly detection , 2020, Neurocomputing.

[65]  Bofeng Zhang,et al.  Deep Edu: A Deep Neural Collaborative Filtering for Educational Services Recommendation , 2020, IEEE Access.

[66]  Ivan Laptev,et al.  Data-driven crowd analysis in videos , 2011, ICCV.

[67]  Nenghai Yu,et al.  Anomaly Detection and Localization: A Novel Two-Phase Framework Based on Trajectory-Level Characteristics , 2018, 2018 IEEE International Conference on Multimedia & Expo Workshops (ICMEW).

[68]  Yonghui Li,et al.  Crowd Scene Analysis by Output Encoding , 2020, ArXiv.

[69]  Luc Van Gool,et al.  Hough Forests for Object Detection, Tracking, and Action Recognition , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[70]  Vishal M. Patel,et al.  CNN-Based cascaded multi-task learning of high-level prior and density estimation for crowd counting , 2017, 2017 14th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS).

[71]  Zarita Zainuddin,et al.  Simulating the Circumambulation of the Ka'aba using SimWalk , 2009 .

[72]  Nuno Vasconcelos,et al.  Bayesian Poisson regression for crowd counting , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[73]  Xuejun Liu,et al.  Real-Time Monitoring for Crowd Counting Using Video Surveillance and GIS , 2012, 2012 2nd International Conference on Remote Sensing, Environment and Transportation Engineering.

[74]  W. Eric L. Grimson,et al.  Adaptive background mixture models for real-time tracking , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[75]  Li Hou,et al.  Image Crowd Counting Using Convolutional Neural Network and Markov Random Field , 2017, J. Adv. Comput. Intell. Intell. Informatics.

[76]  Nuno Vasconcelos,et al.  Modeling, Clustering, and Segmenting Video with Mixtures of Dynamic Textures , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[77]  Christian Oberli,et al.  Crowded pedestrian counting at bus stops from perspective transformations of foreground areas , 2012 .

[78]  Daniel Oñoro-Rubio,et al.  Towards Perspective-Free Object Counting with Deep Learning , 2016, ECCV.

[79]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[80]  Samir A. Elsagheer Mohamed,et al.  Crowd Modeling Based Auto Activated Barriers for Management of Pilgrims in Mataf , 2019, 2019 International Conference on Innovative Trends in Computer Engineering (ITCE).

[81]  Kuldeep Singh,et al.  Convolutional neural networks for crowd behaviour analysis: a survey , 2019, The Visual Computer.

[82]  Areej Al-Wabil,et al.  Multimedia Surveillance in Event Detection: Crowd Analytics in Hajj , 2014, HCI.

[83]  Sultan Daud Khan Congestion detection in pedestrian crowds using oscillation in motion trajectories , 2019, Eng. Appl. Artif. Intell..

[84]  Michael G. Strintzis,et al.  Swarm Intelligence for Detecting Interesting Events in Crowded Environments , 2015, IEEE Transactions on Image Processing.

[85]  Nuno Vasconcelos,et al.  Privacy preserving crowd monitoring: Counting people without people models or tracking , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[86]  Marimuthu Palaniswami,et al.  Anomalous Crowd Event Analysis Using Isometric Mapping , 2015, SIRS.

[87]  Yangsheng Xu,et al.  Crowd Density Estimation Using Texture Analysis and Learning , 2006, 2006 IEEE International Conference on Robotics and Biomimetics.

[88]  Haroon Idrees,et al.  Composition Loss for Counting, Density Map Estimation and Localization in Dense Crowds , 2018, ECCV.

[89]  J. Ferryman,et al.  PETS2009: Dataset and challenge , 2009, 2009 Twelfth IEEE International Workshop on Performance Evaluation of Tracking and Surveillance.

[90]  David A. Forsyth,et al.  Object Recognition as Machine Translation: Learning a Lexicon for a Fixed Image Vocabulary , 2002, ECCV.

[91]  Hichem Snoussi,et al.  Abnormal event detection via the analysis of multi-frame optical flow information , 2019, Frontiers of Computer Science.

[92]  David J. Fleet,et al.  Probabilistic Detection and Tracking of Motion Boundaries , 2000, International Journal of Computer Vision.

[93]  Angel Cruz-Roa,et al.  Mitosis detection in breast cancer pathology images by combining handcrafted and convolutional neural network features , 2014, Journal of medical imaging.

[94]  Ullrich Köthe,et al.  Learning to count with regression forest and structured labels , 2012, Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012).

[95]  Hugo De Klee,et al.  Kingdom of Saudi Arabia , 2000, Research, Innovation and Entrepreneurship in Saudi Arabia.

[96]  Jaya Dewan,et al.  Abnormal Crowd Behavior Detection Using Image Processing , 2018, 2018 Fourth International Conference on Computing Communication Control and Automation (ICCUBEA).

[97]  L. Duczmal,et al.  Nonparametric intensity bounds for the delineation of spatial clusters , 2011, International journal of health geographics.

[98]  Frédéric Jurie,et al.  Fast Discriminative Visual Codebooks using Randomized Clustering Forests , 2006, NIPS.

[99]  Francisco Herrera,et al.  Revisiting crowd behaviour analysis through deep learning: Taxonomy, anomaly detection, crowd emotions, datasets, opportunities and prospects , 2020, Information Fusion.

[100]  Shaogang Gong,et al.  Feature Mining for Localised Crowd Counting , 2012, BMVC.

[101]  Ramakant Nevatia,et al.  Detection of multiple, partially occluded humans in a single image by Bayesian combination of edgelet part detectors , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[102]  Eduard Ayguadé,et al.  Task-Based Crowd Simulation for Heterogeneous Architectures , 2016 .

[103]  Arindam Sikdar,et al.  An adaptive training-less framework for anomaly detection in crowd scenes , 2020, Neurocomputing.

[104]  Masitah Ghazali,et al.  Integrating Perception into V − Hajj: 3D Tawaf Training Simulation Application , 2011 .

[105]  Lior Wolf,et al.  Learning to Count with CNN Boosting , 2016, ECCV.

[106]  Greg Mori,et al.  Detecting Pedestrians by Learning Shapelet Features , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[107]  Yan Wang,et al.  Dense crowd counting from still images with convolutional neural networks , 2016, J. Vis. Commun. Image Represent..

[108]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[109]  Pietro Perona,et al.  Pedestrian Detection: An Evaluation of the State of the Art , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[110]  Salma Kammoun Jarraya,et al.  CNN-Based Crowd Counting Through IoT: Application For Saudi Public Places , 2019, Procedia Computer Science.

[111]  Silvio Savarese,et al.  Tracking the Untrackable: Learning to Track Multiple Cues with Long-Term Dependencies , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[112]  Fei-Yue Wang,et al.  Traffic Flow Prediction With Big Data: A Deep Learning Approach , 2015, IEEE Transactions on Intelligent Transportation Systems.

[113]  Kinjal Mistree,et al.  A review on abnormal crowd behavior detection , 2017, 2017 International Conference on Innovations in Information, Embedded and Communication Systems (ICIIECS).

[114]  Stefan Winkler,et al.  Deep Learning for Emotion Recognition on Small Datasets using Transfer Learning , 2015, ICMI.

[115]  Yu Wang,et al.  Crowd Density Estimation Using Fusion of Multi-Layer Features , 2020, IEEE Transactions on Intelligent Transportation Systems.

[116]  Fabio A. González,et al.  Cascaded ensemble of convolutional neural networks and handcrafted features for mitosis detection , 2014, Medical Imaging.

[117]  Chee Seng Chan,et al.  Crowd Saliency Detection via Global Similarity Structure , 2014, 2014 22nd International Conference on Pattern Recognition.

[118]  Navjot Singh,et al.  KUMBH MELA: a case study for dense crowd counting and modeling , 2020, Multimedia Tools and Applications.

[119]  Duan-Yu Chen,et al.  Visual-Based Human Crowds Behavior Analysis Based on Graph Modeling and Matching , 2013, IEEE Sensors Journal.

[120]  Yadong Mu,et al.  Recurrent Attentive Zooming for Joint Crowd Counting and Precise Localization , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[121]  Xiangjian He,et al.  Performance-enhancing network pruning for crowd counting , 2019, Neurocomputing.

[122]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[123]  Zarita Zainuddin,et al.  Simulation of the Pedestrian Flow in the Tawaf Area Using the Social Force Model , 2010 .

[124]  Robert T. Collins,et al.  Marked point processes for crowd counting , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[125]  Ming Zhu,et al.  Attentive multi-stage convolutional neural network for crowd counting , 2020, Pattern Recognit. Lett..

[126]  Dorothy Ndedi Monekosso,et al.  Detection of Salient Regions in Crowded Scenes , 2014, ArXiv.

[127]  Guoping Qiu,et al.  Crowd density estimation based on rich features and random projection forest , 2016, 2016 IEEE Winter Conference on Applications of Computer Vision (WACV).

[128]  Marta C. González,et al.  Using Convolutional Networks and Satellite Imagery to Identify Patterns in Urban Environments at a Large Scale , 2017, KDD.

[129]  Russ Burtner,et al.  INTERNATIONAL JOURNAL OF HEALTH GEOGRAPHICS REVIEW Open Access , 2022 .

[130]  Khalil Khan,et al.  Automatic Gender Classification through Face Segmentation , 2019, Symmetry.