A Survey of Vision-Based Trajectory Learning and Analysis for Surveillance

This paper presents a survey of trajectory-based activity analysis for visual surveillance. It describes techniques that use trajectory data to define a general set of activities that are applicable to a wide range of scenes and environments. Events of interest are detected by building a generic topographical scene description from underlying motion structure as observed over time. The scene topology is automatically learned and is distinguished by points of interest and motion characterized by activity paths. The methods we review are intended for real-time surveillance through definition of a diverse set of events for further analysis triggering, including virtual fencing, speed profiling, behavior classification, anomaly detection, and object interaction.

[1]  M. Tahar Kechadi,et al.  A Hybrid HMM-SVM Method for Online Handwriting Symbol Recognition , 2006, Sixth International Conference on Intelligent Systems Design and Applications.

[2]  Osama Masoud,et al.  Computer vision algorithms for intersection monitoring , 2003, IEEE Trans. Intell. Transp. Syst..

[3]  Xiaolu Liu,et al.  Development and evaluation of real-time video surveillance system on highway based on semantic hierarchy and decision surface , 2005, 2005 IEEE International Conference on Systems, Man and Cybernetics.

[4]  Alexandre Bernardino,et al.  Detection and classification of highway lanes using vehicle motion trajectories , 2006, IEEE Transactions on Intelligent Transportation Systems.

[5]  Jean-Philippe Thiran,et al.  Multi-Layer Hierarchical Clustering of Pedestrian Trajectories for Automatic Counting of People in Video Sequences , 2005, 2005 Seventh IEEE Workshops on Applications of Computer Vision (WACV/MOTION'05) - Volume 1.

[6]  Zhihai He,et al.  Low-Complexity and Reliable Moving Objects Detection and Tracking for Aerial Video Surveillance with Small UAVS , 2007, 2007 IEEE International Symposium on Circuits and Systems.

[7]  T. Warren Liao,et al.  Clustering of time series data - a survey , 2005, Pattern Recognit..

[8]  W. Eric L. Grimson,et al.  Adaptive background mixture models for real-time tracking , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[9]  Chih-Wen Su,et al.  Real-time event detection and its application to surveillance systems , 2006, 2006 IEEE International Symposium on Circuits and Systems.

[10]  Mubarak Shah,et al.  Machine Vision and Applications Understanding Human Behavior from Motion Imagery , 2003 .

[11]  Fatih Murat Porikli,et al.  Learning object trajectory patterns by spectral clustering , 2004, 2004 IEEE International Conference on Multimedia and Expo (ICME) (IEEE Cat. No.04TH8763).

[12]  Jianbo Shi,et al.  Detecting unusual activity in video , 2004, CVPR 2004.

[13]  Zhouyu Fu,et al.  Semantic-Based Surveillance Video Retrieval , 2007, IEEE Transactions on Image Processing.

[14]  Tarak Gandhi,et al.  Vehicle Surround Capture: Survey of Techniques and a Novel Omni-Video-Based Approach for Dynamic Panoramic Surround Maps , 2006, IEEE Transactions on Intelligent Transportation Systems.

[15]  Tieniu Tan,et al.  Traffic accident prediction using 3-D model-based vehicle tracking , 2004, IEEE Transactions on Vehicular Technology.

[16]  Robert T. Collins,et al.  Articulated Motion Modeling for Activity Analysis , 2004, CVPR 2004.

[17]  Tieniu Tan,et al.  A hierarchical self-organizing approach for learning the patterns of motion trajectories , 2004, IEEE Trans. Neural Networks.

[18]  Marcello Pelillo,et al.  Dominant sets and hierarchical clustering , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[19]  Chris Stauffer,et al.  Estimating Tracking Sources and Sinks , 2003, 2003 Conference on Computer Vision and Pattern Recognition Workshop.

[20]  Stefano Messelodi,et al.  A computer vision system for the detection and classification of vehicles at urban road intersections , 2005, Pattern Analysis and Applications.

[21]  Wei Hu,et al.  A Coarse-to-Fine Strategy for Vehicle Motion Trajectory Clustering , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[22]  Andrew J. Bulpitt,et al.  Learning spatio-temporal patterns for predicting object behaviour , 2000, Image Vis. Comput..

[23]  Mubarak Shah,et al.  Tracking across multiple cameras with disjoint views , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[24]  Jun-Wei Hsieh,et al.  Automatic traffic surveillance system for vehicle tracking and classification , 2006, IEEE Transactions on Intelligent Transportation Systems.

[25]  Jake K. Aggarwal,et al.  Hierarchical Recognition of Human Activities Interacting with Objects , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[26]  Shehzad Khalid,et al.  Motion Trajectory Learning in the DFT-Coefficient Feature Space , 2006, Fourth IEEE International Conference on Computer Vision Systems (ICVS'06).

[27]  Ching-Yao Chan,et al.  Defining Safety Performance Measures of Driver-Assistance Systems for Intersection Left-Turn Conflicts , 2006, 2006 IEEE Intelligent Vehicles Symposium.

[28]  Tarak Gandhi,et al.  Multi-camera Based Traffic Flow Characterization & Classification , 2007, 2007 IEEE Intelligent Transportation Systems Conference.

[29]  Tarek Sayed,et al.  Probabilistic Collision Prediction for Vision-Based Automated Road Safety Analysis , 2007, 2007 IEEE Intelligent Transportation Systems Conference.

[30]  Jake K. Aggarwal,et al.  A hierarchical Bayesian network for event recognition of human actions and interactions , 2004, Multimedia Systems.

[31]  Tieniu Tan,et al.  Comparison of Similarity Measures for Trajectory Clustering in Outdoor Surveillance Scenes , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[32]  Gian Luca Foresti,et al.  On-line trajectory clustering for anomalous events detection , 2006, Pattern Recognit. Lett..

[33]  Osama Masoud,et al.  Learning Traffic Patterns at Intersections by Spectral Clustering of Motion Trajectories , 2006, 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[34]  Shaogang Gong,et al.  Video behaviour profiling and abnormality detection without manual labelling , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[35]  Dimitrios Gunopulos,et al.  Discovering similar multidimensional trajectories , 2002, Proceedings 18th International Conference on Data Engineering.

[36]  Rangachar Kasturi,et al.  Activity recognition based on multiple motion trajectories , 2004, ICPR 2004.

[37]  Tarak Gandhi,et al.  Distributed interactive video arrays for event capture and enhanced situational awareness , 2005, IEEE Intelligent Systems.

[38]  Paolo Remagnino,et al.  Distributed intelligence for multi-camera visual surveillance , 2004, Pattern Recognit..

[39]  Biing-Hwang Juang,et al.  Fundamentals of speech recognition , 1993, Prentice Hall signal processing series.

[40]  Tony Jebara,et al.  Probability Product Kernels , 2004, J. Mach. Learn. Res..

[41]  Ian D. Reid,et al.  Behaviour understanding in video: a combined method , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[42]  Dan Schonfeld,et al.  HMM-based motion recognition system using segmented PCA , 2005, IEEE International Conference on Image Processing 2005.

[43]  W. Eric L. Grimson,et al.  Learning Patterns of Activity Using Real-Time Tracking , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[44]  Michal Irani,et al.  Detecting Irregularities in Images and in Video , 2005, ICCV.

[45]  Tieniu Tan,et al.  Learning activity patterns using fuzzy self-organizing neural network , 2004, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[46]  George Kollios,et al.  Extraction and clustering of motion trajectories in video , 2004, Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004..

[47]  Mohan M. Trivedi,et al.  Multi-person interaction and activity analysis: a synergistic track- and body-level analysis framework , 2007, Machine Vision and Applications.

[48]  W. Eric L. Grimson,et al.  Unsupervised Activity Perception by Hierarchical Bayesian Models , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[49]  Nelson H. C. Yung,et al.  Vehicle-type identification through automated virtual loop assignment and block-based direction biased motion estimation , 1999, Proceedings 199 IEEE/IEEJ/JSAI International Conference on Intelligent Transportation Systems (Cat. No.99TH8383).

[50]  Kuntal Sengupta,et al.  Framework for real-time behavior interpretation from traffic video , 2005, IEEE Transactions on Intelligent Transportation Systems.

[51]  Pavel Berkhin,et al.  A Survey of Clustering Data Mining Techniques , 2006, Grouping Multidimensional Data.

[52]  Mohan M. Trivedi,et al.  Learning and Classification of Trajectories in Dynamic Scenes: A General Framework for Live Video Analysis , 2008, 2008 IEEE Fifth International Conference on Advanced Video and Signal Based Surveillance.

[53]  Mohan M. Trivedi,et al.  Low-Level Segmentation of Aerial Images with Fuzzy Clustering , 1986, IEEE Transactions on Systems, Man, and Cybernetics.

[54]  David C. Hogg,et al.  Learning the distribution of object trajectories for event recognition , 1996, Image Vis. Comput..

[55]  Tim J. Ellis,et al.  Learning semantic scene models from observing activity in visual surveillance , 2005, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[56]  David A. Forsyth,et al.  Learning the Behavior of Users in a Public Space through Video Tracking , 2005, 2005 Seventh IEEE Workshops on Applications of Computer Vision (WACV/MOTION'05) - Volume 1.

[57]  Wolfram Burgard,et al.  Utilizing Learned Motion Patterns to Robustly Track Persons , 2003 .

[58]  Mohan M. Trivedi,et al.  Novel concepts and challenges for the next generation of video surveillance systems , 2007, Machine Vision and Applications.

[59]  Osama Masoud,et al.  A vision-based approach to collision prediction at traffic intersections , 2005, IEEE Transactions on Intelligent Transportation Systems.

[60]  Mohan M. Trivedi,et al.  Learning, Modeling, and Classification of Vehicle Track Patterns from Live Video , 2008, IEEE Transactions on Intelligent Transportation Systems.

[61]  Tieniu Tan,et al.  A system for learning statistical motion patterns , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[62]  Katsushi Ikeuchi,et al.  Traffic monitoring and accident detection at intersections , 2000, IEEE Trans. Intell. Transp. Syst..

[63]  Tieniu Tan,et al.  A survey on visual surveillance of object motion and behaviors , 2004, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[64]  Dietmar Bauer,et al.  Track-Based Finding of Stopping Pedestrians - A Practical Approach for Analyzing a Public Infrastructure , 2006, 2006 IEEE Intelligent Transportation Systems Conference.

[65]  Tim J. Ellis,et al.  Path detection in video surveillance , 2002, Image Vis. Comput..

[66]  Neil Robertson,et al.  Behaviour Recognition and Explanation for Video Surveillance , 2006 .

[67]  Vladimir Kolmogorov,et al.  An Experimental Comparison of Min-Cut/Max-Flow Algorithms for Energy Minimization in Vision , 2004, IEEE Trans. Pattern Anal. Mach. Intell..

[68]  Osama Masoud,et al.  Detection of loitering individuals in public transportation areas , 2005, IEEE Transactions on Intelligent Transportation Systems.

[69]  A. Fascioli,et al.  Pedestrian Protection Systems : Issues , Survey , and Challenges , 2007 .

[70]  Yo-Sung Ho,et al.  Traffic parameter extraction using video-based vehicle tracking , 1999, Proceedings 199 IEEE/IEEJ/JSAI International Conference on Intelligent Transportation Systems (Cat. No.99TH8383).

[71]  Tim J. Ellis,et al.  Bridging the gaps between cameras , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[72]  James Orwell,et al.  Learning the Semantic Landscape: embedding scene knowledge in object tracking , 2005, Real Time Imaging.

[73]  Mubarak Shah,et al.  Multi feature path modeling for video surveillance , 2004, Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004..

[74]  Anil K. Jain,et al.  Data clustering: a review , 1999, CSUR.

[75]  David C. Hogg,et al.  On the feasibility of using a cognitive model to filter surveillance data , 2005, IEEE Conference on Advanced Video and Signal Based Surveillance, 2005..

[76]  Osama Masoud,et al.  A collision prediction system for traffic intersections , 2005, 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[77]  Hilary Buxton,et al.  Learning and understanding dynamic scene activity: a review , 2003, Image Vis. Comput..

[78]  Anil K. Jain,et al.  Landscape of clustering algorithms , 2004, ICPR 2004.

[79]  Matthew Brand,et al.  Discovery and Segmentation of Activities in Video , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[80]  Dan Schonfeld,et al.  Object Trajectory-Based Activity Classification and Recognition Using Hidden Markov Models , 2007, IEEE Transactions on Image Processing.

[81]  Ramakant Nevatia,et al.  Large-scale event detection using semi-hidden Markov models , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[82]  Andrew Hunter,et al.  Application of the self-organising map to trajectory classification , 2000, Proceedings Third IEEE International Workshop on Visual Surveillance.

[83]  Daniel J. Dailey,et al.  Dynamic camera calibration of roadside traffic management cameras for vehicle speed estimation , 2003, IEEE Trans. Intell. Transp. Syst..

[84]  Teuvo Kohonen,et al.  The self-organizing map , 1990 .

[85]  Edward Y. Chang,et al.  Anatomy of a multicamera video surveillance system , 2004, Multim. Syst..

[86]  Marcello Pelillo,et al.  Dominant Sets and Pairwise Clustering , 2007 .

[87]  Dimitrios Gunopulos,et al.  Iterative Incremental Clustering of Time Series , 2004, EDBT.

[88]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[89]  Shehzad Khalid,et al.  Classifying spatiotemporal object trajectories using unsupervised learning in the coefficient feature space , 2006, Multimedia Systems.