Discovering the Top-k Unexplained Sequences in Time-Stamped Observation Data

There are numerous applications where we wish to discover unexpected activities in a sequence of time-stamped observation data-for instance, we may want to detect inexplicable events in transactions at a website or in video of an airport tarmac. In this paper, we start with a known set A of activities (both innocuous and dangerous) that we wish to monitor. However, in addition, we wish to identify “unexplained” subsequences in an observation sequence that are poorly explained (e.g., because they may contain occurrences of activities that have never been seen or anticipated before, i.e., they are not in A). We formally define the probability that a sequence of observations is unexplained (totally or partially) w.r.t. A. We develop efficient algorithms to identify the top-k Totally and partially unexplained sequences w.r.t. A. These algorithms leverage theorems that enable us to speed up the search for totally/partially unexplained sequences. We describe experiments using real-world video and cyber-security data sets showing that our approach works well in practice in terms of both running time and accuracy.

[1]  Shuji Tsukiyama,et al.  A New Algorithm for Generating All the Maximal Independent Sets , 1977, SIAM J. Comput..

[2]  Ehud Rivlin,et al.  Robust Real-Time Unusual Event Detection using Multiple Fixed-Location Monitors , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[3]  Hongli Zhang,et al.  IDS alerts correlation using grammar-based approach , 2009, Journal in Computer Virology.

[4]  Larry S. Davis,et al.  Understanding videos, constructing plots learning a visually grounded storyline model from annotated videos , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[5]  V. S. Subrahmanian,et al.  Detecting Stochastically Scheduled Activities in Video , 2007, IJCAI.

[6]  Aggelos K. Katsaggelos,et al.  Video anomaly detection in spatiotemporal context , 2010, 2010 IEEE International Conference on Image Processing.

[7]  Yan Huang,et al.  ARGMode - Activity Recognition using Graphical Models , 2003, 2003 Conference on Computer Vision and Pattern Recognition Workshop.

[8]  Anthony Hoogs,et al.  Detecting rare events in video using semantic primitives with HMM , 2004, Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004..

[9]  Wenke Lee,et al.  Statistical Causality Analysis of INFOSEC Alert Data , 2003, RAID.

[10]  Gabriel Maciá-Fernández,et al.  Anomaly-based network intrusion detection: Techniques, systems and challenges , 2009, Comput. Secur..

[11]  Alessandro Mecocci,et al.  A completely autonomous system that learns anomalous movements in advanced videosurveillance applications , 2005, IEEE International Conference on Image Processing 2005.

[12]  Alex Pentland,et al.  Coupled hidden Markov models for complex action recognition , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[13]  Shaogang Gong,et al.  Video Behavior Profiling for Anomaly Detection , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  Rama Chellappa,et al.  "Shape Activity": a continuous-state HMM for moving/deforming shapes with application to abnormal activity detection , 2005, IEEE Transactions on Image Processing.

[15]  Eric Horvitz,et al.  Layered representations for human activity recognition , 2002, Proceedings. Fourth IEEE International Conference on Multimodal Interfaces.

[16]  Rama Chellappa,et al.  Activity Modeling Using Event Probability Sequences , 2008, IEEE Transactions on Image Processing.

[17]  Nipun Kwatra,et al.  A Framework for Activity Recognition and Detection of Unusual Activities , 2004, ICVGIP.

[18]  James J. Clark,et al.  Anomaly Detection for Video Surveillance Applications , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[19]  Xinzhou Qin,et al.  A Probabilistic-Based Framework for INFOSEC Alert Correlation , 2005 .

[20]  Fabio Persia,et al.  Finding "Unexplained" Activities in Video , 2011, IJCAI.

[21]  Mohan M. Trivedi,et al.  Trajectory Learning for Activity Understanding: Unsupervised, Multilevel, and Long-Term Adaptive Approach , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[22]  Junbo Wang,et al.  Design of a Situation-Aware System for Abnormal Activity Detection of Elderly People , 2012, AMT.

[23]  Sushil Jajodia,et al.  Using attack graphs for correlating, hypothesizing, and predicting intrusion alerts , 2006, Comput. Commun..

[24]  Todd L. Heberlein,et al.  Network intrusion detection , 1994, IEEE Network.

[25]  Samy Bengio,et al.  Semi-supervised adapted HMMs for unusual event detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[26]  Ramakant Nevatia,et al.  Multi-agent event recognition , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[27]  Aggelos K. Katsaggelos,et al.  Detecting contextual anomalies of crowd motion in surveillance video , 2009, 2009 16th IEEE International Conference on Image Processing (ICIP).

[28]  Hervé Debar,et al.  Aggregation and Correlation of Intrusion-Detection Alerts , 2001, Recent Advances in Intrusion Detection.

[29]  Jianbo Shi,et al.  Detecting unusual activity in video , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[30]  Yang Gao,et al.  Detecting Abnormal Events via Hierarchical Dirichlet Processes , 2009, PAKDD.

[31]  Sushil Jajodia,et al.  Scalable Analysis of Attack Scenarios , 2011, ESORICS.

[32]  Song Li,et al.  Temporal signatures for intrusion detection , 2001, Seventeenth Annual Computer Security Applications Conference.

[33]  Alessia Saggese,et al.  A Clustering Algorithm of Trajectories for Behaviour Understanding Based on String Kernels , 2012, 2012 Eighth International Conference on Signal Image Technology and Internet Based Systems.

[34]  Peng Ning,et al.  Constructing attack scenarios through correlation of intrusion alerts , 2002, CCS '02.

[35]  Shuicheng Yan,et al.  Detecting Anomaly in Videos from Trajectory Similarity Analysis , 2007, 2007 IEEE International Conference on Multimedia and Expo.

[36]  Qiang Yang,et al.  Sensor-Based Abnormal Human-Activity Detection , 2008, IEEE Transactions on Knowledge and Data Engineering.

[37]  Sushil Jajodia,et al.  Correlating intrusion events and building attack scenarios through attack graph distances , 2004, 20th Annual Computer Security Applications Conference.

[38]  Alexander Aiken,et al.  Community Epidemic Detection Using Time-Correlated Anomalies , 2010, RAID.

[39]  K. Grauman,et al.  Observe locally, infer globally: A space-time MRF for detecting abnormal activities with incremental updates , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.