PADUA: Parallel Architecture to Detect Unexplained Activities

There are numerous applications (e.g., video surveillance, fraud detection, cybersecurity) in which we wish to identify unexplained sets of events. Most related past work has been domain-dependent (e.g., video surveillance, cybersecurity) and has focused on the valuable class of statistical anomalies in which statistically unusual events are considered. In contrast, suppose there is a set A of known activity models (both harmless and harmful) and a log L of time-stamped observations. We define a part L'⊆ L of the log to represent an unexplained situation when none of the known activity models can explain L' with a score exceeding a user-specified threshold. We represent activities via probabilistic penalty graphs (PPGs) and show how a set of PPGs can be combined into one Super-PPG for which we define an index structure. Given a compute cluster of (K + 1) nodes (one of which is a master node), we show how to split a Super-PPG into K subgraphs, each of which can be independently processed by a compute node. We provide algorithms for the individual compute nodes to ensure seamless handoffs that maximally leverage parallelism. PADUA is domain-independent and can be applied to many domains (perhaps with some specialization). We conducted detailed experiments with PADUA on two real-world datasets—the ITEA CANDELA video surveillance dataset and a network traffic dataset appropriate for cybersecurity applications. PADUA scales extremely well with the number of processors and significantly outperforms past work both in accuracy and time. Thus, PADUA represents the first parallel architecture and algorithm for identifying unexplained situations in observation data, offering both scalability and accuracy.

[1]  Aggelos K. Katsaggelos,et al.  Detecting contextual anomalies of crowd motion in surveillance video , 2009, 2009 16th IEEE International Conference on Image Processing (ICIP).

[2]  Alessia Saggese,et al.  A Clustering Algorithm of Trajectories for Behaviour Understanding Based on String Kernels , 2012, 2012 Eighth International Conference on Signal Image Technology and Internet Based Systems.

[3]  Gabriel Maciá-Fernández,et al.  Anomaly-based network intrusion detection: Techniques, systems and challenges , 2009, Comput. Secur..

[4]  Ivandro Sanches.  Noise-compensated hidden Markov models , 2000, IEEE Trans. Speech Audio Process..

[5]  Todd L. Heberlein,et al.  Network intrusion detection , 1994, IEEE Network.

[6]  Shaogang Gong,et al.  Video Behavior Profiling for Anomaly Detection , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[7]  Xinzhou Qin,et al.  A Probabilistic-Based Framework for INFOSEC Alert Correlation , 2005 .

[8]  Ricardo Vilalta,et al.  Predicting rare events in temporal domains , 2002, 2002 IEEE International Conference on Data Mining, 2002. Proceedings..

[9]  Zoubin Ghahramani,et al.  Learning Dynamic Bayesian Networks , 1997, Summer School on Neural Networks.

[10]  Samy Bengio,et al.  Semi-supervised adapted HMMs for unusual event detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[11]  Eric Horvitz,et al.  Layered representations for human activity recognition , 2002, Proceedings. Fourth IEEE International Conference on Multimodal Interfaces.

[12]  Rama Chellappa,et al.  Activity Modeling Using Event Probability Sequences , 2008, IEEE Transactions on Image Processing.

[13]  Nipun Kwatra,et al.  A Framework for Activity Recognition and Detection of Unusual Activities , 2004, ICVGIP.

[14]  Fabio Persia,et al.  Finding "Unexplained" Activities in Video , 2011, IJCAI.

[15]  Junbo Wang,et al.  Design of a Situation-Aware System for Abnormal Activity Detection of Elderly People , 2012, AMT.

[16]  James J. Clark,et al.  Anomaly Detection for Video Surveillance Applications , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[17]  Azaria Paz,et al.  Probabilistic automata , 2003 .

[18]  Sushil Jajodia,et al.  Using attack graphs for correlating, hypothesizing, and predicting intrusion alerts , 2006, Comput. Commun..

[19]  P. Lancaster.  Curve and surface fitting , 1986 .

[20]  Ramakant Nevatia,et al.  Video-based event recognition: activity representation and probabilistic recognition methods , 2004, Comput. Vis. Image Underst..

[21]  V. S. Subrahmanian,et al.  Detecting Stochastically Scheduled Activities in Video , 2007, IJCAI.

[22]  Jianbo Shi,et al.  Detecting unusual activity in video , 2004, CVPR 2004.

[23]  Yang Gao,et al.  Detecting Abnormal Events via Hierarchical Dirichlet Processes , 2009, PAKDD.

[24]  Kristen Grauman,et al.  Observe locally, infer globally: A space-time MRF for detecting abnormal activities with incremental updates , 2009, CVPR.

[25]  Ehud Rivlin,et al.  Robust Real-Time Unusual Event Detection using Multiple Fixed-Location Monitors , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[26]  Hongli Zhang,et al.  IDS alerts correlation using grammar-based approach , 2009, Journal in Computer Virology.

[27]  Michael Philippsen,et al.  Learning event detection rules with noise hidden Markov models , 2012, 2012 NASA/ESA Conference on Adaptive Hardware and Systems (AHS).

[28]  Ramakrishnan Srikant,et al.  Fast Algorithms for Mining Association Rules in Large Databases , 1994, VLDB.

[29]  Wenke Lee,et al.  Statistical Causality Analysis of INFOSEC Alert Data , 2003, RAID.

[30]  Alex Pentland,et al.  Coupled hidden Markov models for complex action recognition , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[31]  Rui Xu,et al.  Survey of clustering algorithms , 2005, IEEE Transactions on Neural Networks.

[32]  Qiang Yang,et al.  Sensor-Based Abnormal Human-Activity Detection , 2008, IEEE Transactions on Knowledge and Data Engineering.

[33]  B. Stefano,et al.  Insurance fraud evaluation: a fuzzy expert system , 2001, 10th IEEE International Conference on Fuzzy Systems. (Cat. No.01CH37297).

[34]  Alexander Aiken,et al.  Community Epidemic Detection Using Time-Correlated Anomalies , 2010, RAID.

[35]  Rama Chellappa,et al.  "Shape Activity": a continuous-state HMM for moving/deforming shapes with application to abnormal activity detection , 2005, IEEE Transactions on Image Processing.

[36]  Reggio Emilia,et al.  Insurance Fraud Evaluation - A Fuzzy Expert System , 2001, FUZZ-IEEE.

[37]  Alexander Artikis,et al.  Behaviour Recognition using the Event Calculus , 2009, AIAI.

[38]  Christel Baier,et al.  Probabilistic ω-automata , 2012, JACM.

[39]  Malik Ghallab,et al.  On Chronicles: Representation, On-line Recognition and Learning , 1996, KR.

[40]  Alessandro Mecocci,et al.  A completely autonomous system that learns anomalous movements in advanced videosurveillance applications , 2005, IEEE International Conference on Image Processing 2005.

[41]  Peng Ning,et al.  Constructing attack scenarios through correlation of intrusion alerts , 2002, CCS '02.

[42]  Haym Hirsh,et al.  Learning to Predict Rare Events in Event Sequences , 1998, KDD.

[43]  Shuicheng Yan,et al.  Detecting Anomaly in Videos from Trajectory Similarity Analysis , 2007, 2007 IEEE International Conference on Multimedia and Expo.

[44]  Peter Lancaster,et al.  Curve and surface fitting - an introduction , 1986 .

[45]  Aggelos K. Katsaggelos,et al.  Video anomaly detection in spatiotemporal context , 2010, 2010 IEEE International Conference on Image Processing.

[46]  Yan Huang,et al.  ARGMode - Activity Recognition using Graphical Models , 2003, 2003 Conference on Computer Vision and Pattern Recognition Workshop.

[47]  David R. Karger,et al.  A new approach to the minimum cut problem , 1996, JACM.

[48]  L. Baum,et al.  Statistical Inference for Probabilistic Functions of Finite State Markov Chains , 1966 .

[49]  Sushil Jajodia,et al.  Scalable Analysis of Attack Scenarios , 2011, ESORICS.

[50]  Song Li,et al.  Temporal signatures for intrusion detection , 2001, Seventeenth Annual Computer Security Applications Conference.

[51]  Girish Keshav Palshikar,et al.  Collusion set detection using graph clustering , 2008, Data Mining and Knowledge Discovery.