Video-Based Abnormal Human Behavior Recognition—A Review

Modeling human behaviors and activity patterns for recognition or detection of special event has attracted significant research interest in recent years. Diverse methods that are abound for building intelligent vision systems aimed at scene understanding and making correct semantic inference from the observed dynamics of moving targets. Most applications are in surveillance, video content retrieval, and human-computer interfaces. This paper presents not only an update extending previous related surveys, but also a focus on contextual abnormal human behavior detection especially in video surveillance applications. The main purpose of this survey is to extensively identify existing methods and characterize the literature in a manner that brings key challenges to attention.

[1]  Sergio A. Velastin,et al.  Intelligent distributed surveillance systems: a review , 2005 .

[2]  Runsheng Wang,et al.  The abnormal behavior analysis of single person on the road based on region and behavior features , 2007, International Symposium on Multispectral Image Processing and Pattern Recognition.

[3]  Jean Meunier,et al.  Fall Detection from Human Shape and Motion History Using Video Surveillance , 2007, 21st International Conference on Advanced Information Networking and Applications Workshops (AINAW'07).

[4]  Yangsheng Xu,et al.  A detection system for human abnormal behavior , 2005, 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[5]  Rama Chellappa,et al.  Statistical shape theory for activity modeling , 2003, 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings. (ICASSP '03)..

[6]  Venkatesh Saligrama,et al.  Motion segmentation and abnormal behavior detection via behavior clustering , 2008, 2008 15th IEEE International Conference on Image Processing.

[7]  Vladimir Pavlovic,et al.  Visual Interpretation of Hand Gestures for Human-Computer Interaction: A Review , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[8]  Lawrence Carin,et al.  Infinite Hidden Markov Models for Unusual-Event Detection in Video , 2008, IEEE Transactions on Image Processing.

[9]  Dariu Gavrila,et al.  The Visual Analysis of Human Movement: A Survey , 1999, Comput. Vis. Image Underst..

[10]  Huosheng Hu,et al.  Human motion tracking for rehabilitation - A survey , 2008, Biomed. Signal Process. Control..

[11]  Victoria J. Hodge,et al.  A Survey of Outlier Detection Methodologies , 2004, Artificial Intelligence Review.

[12]  Shaogang Gong,et al.  Global Behaviour Inference using Probabilistic Latent Semantic Analysis , 2008, BMVC.

[13]  Jesse Hoey,et al.  Automated Detection of Unusual Events on Stairs , 2006, The 3rd Canadian Conference on Computer and Robot Vision (CRV'06).

[14]  Hui-min Qian,et al.  SVM-based abnormal activity detection for home care , 2008, 2008 7th World Congress on Intelligent Control and Automation.

[15]  Dmitry B. Goldgof,et al.  Understanding Transit Scenes: A Survey on Human Behavior-Recognition Algorithms , 2010, IEEE Transactions on Intelligent Transportation Systems.

[16]  Ye Zhang,et al.  Irregular behavior recognition based on treading track , 2007, 2007 International Conference on Wavelet Analysis and Pattern Recognition.

[17]  Adrian Hilton,et al.  A survey of advances in vision-based human motion capture and analysis , 2006, Comput. Vis. Image Underst..

[18]  Aaron F. Bobick,et al.  Recognition of multi-agent interaction in video surveillance , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[19]  Rama Chellappa,et al.  Machine Recognition of Human Activities: A Survey , 2008, IEEE Transactions on Circuits and Systems for Video Technology.

[20]  Jun Zhang,et al.  Anomalous Detection Based on Adaboost-HMM , 2006, 2006 6th World Congress on Intelligent Control and Automation.

[21]  Tim J. Ellis,et al.  Learning semantic scene models from observing activity in visual surveillance , 2005, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[22]  Tieniu Tan,et al.  A system for learning statistical motion patterns , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[23]  Christophe Rosenberger,et al.  Abnormal events detection based on spatio-temporal co-occurences , 2009, CVPR.

[24]  M.M. Deris,et al.  A Comparative Study for Outlier Detection Techniques in Data Mining , 2006, 2006 IEEE Conference on Cybernetics and Intelligent Systems.

[25]  Luc Van Gool,et al.  Exploiting simple hierarchies for unsupervised human behavior analysis , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[26]  Pau-Choo Chung,et al.  A daily behavior enabled hidden Markov model for human behavior understanding , 2008, Pattern Recognit..

[27]  Shuicheng Yan,et al.  Detecting Anomaly in Videos from Trajectory Similarity Analysis , 2007, 2007 IEEE International Conference on Multimedia and Expo.

[28]  Alberto Del Bimbo,et al.  Effective Codebooks for human action categorization , 2009, 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops.

[29]  Fred A. Hamprecht,et al.  Weakly Supervised Learning of a Classifier for Unusual Event Detection , 2008, IEEE Transactions on Image Processing.

[30]  Rita Cucchiara.  When multimedia meets surveillance and forensics in people security , 2010, MiFor '10.

[31]  Mubarak Shah,et al.  A survey of motion analysis from moving light displays , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[32]  Jun Zhang,et al.  Detecting Pedestrian Abnormal Behavior Based on Fuzzy Associative Memory , 2008, 2008 Fourth International Conference on Natural Computation.

[33]  Gaurav S. Sukhatme,et al.  Detecting anomalous human interactions using laser range-finders , 2004, 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Cat. No.04CH37566).

[34]  Takashi Omori,et al.  Detection of unusual human behavior in intelligent house , 2002, Proceedings of the 12th IEEE Workshop on Neural Networks for Signal Processing.

[35]  Takeo Kanade,et al.  Introduction to the Special Section on Video Surveillance , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[36]  Angelo Chianese,et al.  Detecting abnormal activities in video sequences , 2008, AMDIT '08.

[37]  Jake K. Aggarwal,et al.  Articulated and elastic non-rigid motion: a review , 1994, Proceedings of 1994 IEEE Workshop on Motion of Non-rigid and Articulated Objects.

[38]  Gian Luca Foresti,et al.  Anomalous trajectory detection using support vector machines , 2007, 2007 IEEE Conference on Advanced Video and Signal Based Surveillance.

[39]  Wei Wei,et al.  Vision-Based Human Motion Recognition: A Survey , 2009, 2009 Second International Conference on Intelligent Networks and Intelligent Systems.

[40]  David Gerónimo Gómez,et al.  Survey of Pedestrian Detection for Advanced Driver Assistance Systems , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[41]  Yangsheng Xu,et al.  Abnormal Behavior Detection by Multi-SVM-Based Bayesian Network , 2007, 2007 International Conference on Information Acquisition.

[42]  Svetha Venkatesh,et al.  Explicit State Duration HMM for Abnormality Detection in Sequences of Human Activity , 2004, PRICAI.

[43]  Sameer Singh,et al.  Novelty detection: a review - part 1: statistical approaches , 2003, Signal Process..

[44]  Jake K. Aggarwal,et al.  Human Motion Analysis: A Review , 1999, Comput. Vis. Image Underst..

[45]  Thomas B. Moeslund,et al.  A Survey of Computer Vision-Based Human Motion Capture , 2001, Comput. Vis. Image Underst..

[46]  Jun Zhang,et al.  Detecting Irregularities by Image Contour Based on Fuzzy Neural Network , 2008, 2008 3rd International Conference on Innovative Computing Information and Control.

[47]  W. Eric L. Grimson,et al.  Unsupervised Activity Perception in Crowded and Complicated Scenes Using Hierarchical Bayesian Models , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[48]  Shaogang Gong,et al.  A Markov Clustering Topic Model for mining behaviour in video , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[49]  Honghai Liu,et al.  Advances in View-Invariant Human Motion Analysis: A Review , 2010, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[50]  Hanning Zhou,et al.  Unusual Event Detection via Multi-camera Video Mining , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[51]  SkubicMarjorie,et al.  Linguistic summarization of video for fall detection using voxel person and fuzzy logic , 2009 .

[52]  Yi-ping Tang,et al.  Intelligent Video Analysis Technology for Elevator Cage Abnormality Detection in Computer Vision , 2009, 2009 Fourth International Conference on Computer Sciences and Convergence Information Technology.

[53]  Kejun Wang,et al.  Ant-based clustering of visual-words for unsupervised human action recognition , 2010, NaBIC.

[54]  Ankush Mittal,et al.  Study of Robust and Intelligent Surveillance in Visible and Multi-modal Framework , 2007, Informatica.

[55]  Kristen Grauman,et al.  Observe locally, infer globally: A space-time MRF for detecting abnormal activities with incremental updates , 2009, CVPR.

[56]  Gian Luca Foresti,et al.  Surveillance-Oriented Event Detection in Video Streams , 2011, IEEE Intelligent Systems.

[57]  Luc Van Gool,et al.  Tracker trees for unusual event detection , 2009, 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops.

[58]  Michael I. Jordan,et al.  Latent Dirichlet Allocation , 2001, J. Mach. Learn. Res..

[59]  Louis Kratz,et al.  Anomaly detection in extremely crowded scenes using spatio-temporal motion pattern models , 2009, CVPR.

[60]  Svetha Venkatesh,et al.  Activity recognition and abnormality detection with the switching hidden semi-Markov model , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[61]  Sameer Singh,et al.  Novelty detection: a review - part 2: : neural network based approaches , 2003, Signal Process..

[62]  Aggelos K. Katsaggelos,et al.  A Dynamic Hierarchical Clustering Method for Trajectory-Based Unusual Video Event Detection , 2009, IEEE Transactions on Image Processing.

[63]  Simone Calderara,et al.  Detection of abnormal behaviors using a mixture of Von Mises distributions , 2007, 2007 IEEE Conference on Advanced Video and Signal Based Surveillance.

[64]  Christopher W. Geib,et al.  The meaning of action: a review on action recognition and mapping , 2007, Adv. Robotics.

[65]  Dariu Gavrila,et al.  Monocular Pedestrian Detection: Survey and Experiments , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[66]  Bir Bhanu,et al.  Anomalous activity classification in the distributed camera network , 2008, 2008 15th IEEE International Conference on Image Processing.

[67]  Jianbo Shi,et al.  Detecting unusual activity in video , 2004, CVPR 2004.

[68]  Michal Irani,et al.  Detecting Irregularities in Images and in Video , 2005, ICCV.

[69]  Sabu Emmanuel,et al.  Intelligent Video Surveillance for Monitoring Elderly in Home Environments , 2007, 2007 IEEE 9th Workshop on Multimedia Signal Processing.

[70]  Shaogang Gong,et al.  Detecting and quantifying unusual interactions by correlating salient motion , 2005, IEEE Conference on Advanced Video and Signal Based Surveillance, 2005..

[71]  Ehud Rivlin,et al.  Robust Real-Time Unusual Event Detection using Multiple Fixed-Location Monitors , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[72]  Stephen J. McKenna,et al.  Activity summarisation and fall detection in a supportive home environment , 2004, ICPR 2004.

[73]  Deborah Goshorn,et al.  Abnormal behavior-detection using sequential syntactical classification in a network of clustered cameras , 2008, 2008 Second ACM/IEEE International Conference on Distributed Smart Cameras.

[74]  Nadipuram R. Prasad,et al.  Machine intelligence-based decision-making (MIND) for automatic anomaly detection , 2007, SPIE Defense + Commercial Sensing.

[75]  Paulo Cortez,et al.  Prediction of Abnormal Behaviors for Intelligent Video Surveillance Systems , 2007, 2007 IEEE Symposium on Computational Intelligence and Data Mining.

[76]  Jung-Min Park,et al.  An overview of anomaly detection techniques: Existing solutions and latest technological trends , 2007, Comput. Networks.

[77]  James F. O'Brien,et al.  Computational Studies of Human Motion , 2006 .

[78]  Mubarak Shah,et al.  Chaotic invariants of Lagrangian particle trajectories for anomaly detection in crowded scenes , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[79]  Ramakant Nevatia,et al.  Single View Human Action Recognition using Key Pose Matching and Viterbi Path Searching , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[80]  Shaogang Gong,et al.  Action categorization by structural probabilistic latent semantic analysis , 2010, Comput. Vis. Image Underst..

[81]  Feng Pan,et al.  Anomaly detection based-on the regularity of normal behaviors , 2006, 2006 1st International Symposium on Systems and Control in Aerospace and Astronautics.

[82]  Juan Carlos Niebles,et al.  Unsupervised Learning of Human Action Categories , 2006 .

[83]  Graham Coleman,et al.  Detection and explanation of anomalous activities: representing activities as bags of event n-grams , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[84]  Arnold Wiliem,et al.  Detecting Uncommon Trajectories , 2008, 2008 Digital Image Computing: Techniques and Applications.

[85]  Alireza Rezvanian,et al.  Robust Fall Detection Using Human Shape and Multi-class Support Vector Machine , 2008, 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing.

[86]  Hilary Buxton,et al.  Learning and understanding dynamic scene activity: a review , 2003, Image Vis. Comput..

[87]  Lars Niklasson,et al.  An ensemble approach for increased anomaly detection performance in video surveillance data , 2009, 2009 12th International Conference on Information Fusion.

[88]  Jean-Marc Odobez,et al.  Topic models for scene analysis and abnormality detection , 2009, 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops.

[89]  Sameer Singh,et al.  Video analysis of human dynamics - a survey , 2003, Real Time Imaging.

[90]  W. Eric L. Grimson,et al.  Unsupervised Activity Perception by Hierarchical Bayesian Models , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[91]  Yang Gao,et al.  Detecting Abnormal Events via Hierarchical Dirichlet Processes , 2009, PAKDD.

[92]  VARUN CHANDOLA,et al.  Anomaly detection: A survey , 2009, CSUR.

[93]  Yangsheng Xu,et al.  Crowd surveillance using Markov Random Fields , 2008, 2008 IEEE International Conference on Automation and Logistics.

[94]  Li He-Ping,et al.  Behavior Modeling and Abnormality Detection Based on Semi-Supervised Learning Method , 2007 .

[95]  Tieniu Tan,et al.  Recent developments in human motion analysis , 2003, Pattern Recognit..

[96]  Nanna Suryana,et al.  Crowd event detection in computer vision , 2010, 2010 2nd International Conference on Signal Processing Systems.

[97]  Bhavani M. Thuraisingham,et al.  A framework for a video analysis tool for suspicious event detection , 2005, MDM@KDD.

[98]  James J. Clark,et al.  Anomaly Detection for Video Surveillance Applications , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[99]  Baharak Shakeri Aski,et al.  Intelligent video surveillance for monitoring fall detection of elderly in home environments , 2008, 2008 11th International Conference on Computer and Information Technology.

[100]  James M. Keller,et al.  Linguistic summarization of video for fall detection using voxel person and fuzzy logic , 2009, Comput. Vis. Image Underst..

[101]  A. Mecocci,et al.  Automatic detection of anomalous behavioural events for advanced real-time video surveillance , 2003, The 3rd International Workshop on Scientific Use of Submarine Cables and Related Technologies, 2003..

[102]  Junsong Yuan,et al.  Sparse reconstruction cost for abnormal event detection , 2011, CVPR 2011.

[103]  Teddy Ko,et al.  A survey on behavior analysis in video surveillance for homeland security applications , 2008, 2008 37th IEEE Applied Imagery Pattern Recognition Workshop.

[104]  Shaogang Gong,et al.  Learning Rare Behaviours , 2010, ACCV.

[105]  Ying Wang,et al.  Abnormal Activity Recognition in Office Based on R Transform , 2007, 2007 IEEE International Conference on Image Processing.

[106]  Vangelis Metsis,et al.  Abnormal human behavioral pattern detection in assisted living environments , 2010, PETRA '10.

[107]  Youfu Wu,et al.  Anomalous Event Detection Based on Self-Organizing Map for Supermarket Monitoring , 2009, 2009 International Conference on Information Engineering and Computer Science.

[108]  Nuno Vasconcelos,et al.  Anomaly detection in crowded scenes , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[109]  Shaogang Gong,et al.  Video Behavior Profiling for Anomaly Detection , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[110]  Chabane Djeraba,et al.  Motion Pattern Extraction and Event Detection for Automatic Visual Surveillance , 2011, EURASIP J. Image Video Process..

[111]  Alex Pentland,et al.  Human computing and machine understanding of human behavior: a survey , 2006, ICMI '06.

[112]  Peng Zhang,et al.  Abnormal Video Sections Detection Based on Inter-frames Information , 2009, 2009 Third International Conference on Multimedia and Ubiquitous Engineering.

[113]  Sridha Sridharan,et al.  Textures of optical flow for real-time anomaly detection in crowds , 2011, 2011 8th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS).

[114]  A. Fascioli,et al.  Pedestrian Protection Systems : Issues , Survey , and Challenges , 2007 .

[115]  Andrea Cavallaro,et al.  Local Abnormality Detection in Video Using Subspace Learning , 2010, 2010 7th IEEE International Conference on Advanced Video and Signal Based Surveillance.

[116]  Qiang Yang,et al.  Sensor-Based Abnormal Human-Activity Detection , 2008, IEEE Transactions on Knowledge and Data Engineering.

[117]  Ronald Poppe,et al.  Vision-based human motion analysis: An overview , 2007, Comput. Vis. Image Underst..

[118]  Tieniu Tan,et al.  A survey on visual surveillance of object motion and behaviors , 2004, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[119]  Jake K. Aggarwal,et al.  Human motion: modeling and recognition of actions and interactions , 2004, Proceedings. 2nd International Symposium on 3D Data Processing, Visualization and Transmission, 2004. 3DPVT 2004..

[120]  Brian C. Lovell,et al.  Improved anomaly detection in crowded scenes via cell-based analysis of foreground speed, size and texture , 2011, CVPR 2011 WORKSHOPS.

[121]  H. Foroughi,et al.  An eigenspace-based approach for human fall detection using Integrated Time Motion Image and Neural Network , 2008, 2008 9th International Conference on Signal Processing.

[122]  Juan Carlos Niebles,et al.  A Hierarchical Model of Shape and Appearance for Human Action Classification , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[123]  Christophe Rosenberger,et al.  Abnormal events detection based on spatio-temporal co-occurences , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[124]  Samy Bengio,et al.  Semi-supervised adapted HMMs for unusual event detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[125]  Chia-Tai Chan,et al.  Abnormality Detection for Improving Elder's Daily Life Independent , 2010, ICOST.

[126]  Ronald Poppe,et al.  A survey on vision-based human action recognition , 2010, Image Vis. Comput..

[127]  Mubarak Shah,et al.  Motion-based recognition a survey , 1995, Image Vis. Comput..

[128]  Rama Chellappa,et al.  "Shape Activity": a continuous-state HMM for moving/deforming shapes with application to abnormal activity detection , 2005, IEEE Transactions on Image Processing.

[129]  Reda Alhajj,et al.  A comprehensive survey of numeric and symbolic outlier mining techniques , 2006, Intell. Data Anal..

[130]  Shaogang Gong,et al.  Incremental and adaptive abnormal behaviour detection , 2008, Comput. Vis. Image Underst..

[131]  Chung-Lin Huang,et al.  Abnormal Event Detection in Video Using N-cut Clustering , 2006, 2006 International Conference on Intelligent Information Hiding and Multimedia.

[132]  David A. Forsyth,et al.  Computational Studies of Human Motion: Part 1, Tracking and Motion Synthesis , 2005, Found. Trends Comput. Graph. Vis..

[133]  P. S. Sastry,et al.  Abnormal activity detection in video sequences using learnt probability densities , 2003, TENCON 2003. Conference on Convergent Technologies for Asia-Pacific Region.

[134]  Ehud Rivlin,et al.  Understanding Video Events: A Survey of Methods for Automatic Interpretation of Semantic Occurrences in Video , 2009, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[135]  Subhashis Banerjee,et al.  Unusual Activity Analysis Using Video Epitomes and pLSA , 2008, 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing.

[136]  Ramin Mehran,et al.  Abnormal crowd behavior detection using social force model , 2009, CVPR.

[137]  A F Bobick,et al.  Movement, activity and action: the role of knowledge in the perception of motion. , 1997, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[138]  A. Beghdadi,et al.  Abnormal behavior detection using a multi-modal stochastic learning approach , 2008, 2008 International Conference on Intelligent Sensors, Sensor Networks and Information Processing.

[139]  Mubarak Shah,et al.  Multi feature path modeling for video surveillance , 2004, Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004..