Conditional anomaly detection in event streams

Abstract Detecting early enough the anomalous behavior of technical systems facilitates cost savings thanks to avoiding system downtimes, guiding maintenance, or improving performance. The novel framework proposed in this paper processes event streams originating from system monitoring for anomaly detection purposes. Therefore, statistical models characterizing the normal behavior of the monitored system are learned from the events. Instead of having one coarse normal model for all operational states, the proposed framework contains a mechanism for automatically detecting different conditions of the system allowing for fine-tuned models for every condition. The performance of the framework is demonstrated by means of a real-world application, where the log files of a large-scale printing machine are analyzed for anomalies.

[1]  Victoria J. Hodge,et al.  A Survey of Outlier Detection Methodologies , 2004, Artificial Intelligence Review.

[2]  Uwe D. Hanebeck,et al.  Superficial Gaussian Mixture Reduction , 2011, GI-Jahrestagung.

[3]  Lindsay Victoria Allen.  Verification and Anomaly Detection for Event-Based Control of Manufacturing Systems , 2010 .

[4]  Raymond T. Ng,et al.  Distance-based outliers: algorithms and applications , 2000, The VLDB Journal.

[5]  Krishna R. Pattipati,et al.  A look at Gaussian mixture reduction algorithms , 2011, 14th International Conference on Information Fusion.

[6]  S. Chiba,et al.  Dynamic programming algorithm optimization for spoken word recognition , 1978 .

[7]  Masashi Sugiyama,et al.  Change-Point Detection in Time-Series Data by Direct Density-Ratio Estimation , 2009, SDM.

[8]  R. Tibshirani,et al.  The solution path of the generalized lasso , 2010, 1005.1971.

[9]  Josefina Andreasson,et al.  Log-Based Anomaly Detection for System Surveillance , 2015 .

[10]  Guy Lapalme,et al.  A systematic analysis of performance measures for classification tasks , 2009, Inf. Process. Manag..

[11]  Sanjay Ranka,et al.  Conditional Anomaly Detection , 2007, IEEE Transactions on Knowledge and Data Engineering.

[12]  Georg Carle,et al.  Traffic Anomaly Detection Using K-Means Clustering , 2007 .

[13]  R. Tibshirani.  Adaptive piecewise polynomial estimation via trend filtering , 2013, 1304.2986.

[14]  Sridhar Ramaswamy,et al.  Efficient algorithms for mining outliers from large data sets , 2000, SIGMOD '00.

[15]  Uwe D. Hanebeck,et al.  Progressive Gaussian mixture reduction , 2008, 2008 11th International Conference on Information Fusion.

[16]  W. B. Cavnar,et al.  N-gram-based text categorization , 1994 .

[17]  VARUN CHANDOLA,et al.  Anomaly detection: A survey , 2009, CSUR.

[18]  Michèle Basseville,et al.  Detection of abrupt changes: theory and application , 1993 .

[19]  Marie B. Levine,et al.  Automated Event Detection in Space Instruments: A Case Study Using IPEX-2 Data and Support Vector Ma , 2000 .

[20]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[21]  V. A. Epanechnikov.  Non-Parametric Estimation of a Multivariate Probability Density , 1969 .

[22]  Ana Arribas-Gil,et al.  Pairwise dynamic time warping for event data , 2012, Comput. Stat. Data Anal..

[23]  Stephen P. Boyd,et al.  1 Trend Filtering , 2009, SIAM Rev..

[24]  Masashi Sugiyama,et al.  Change-point detection in time-series data by relative density-ratio estimation , 2012 .

[25]  Matthew P. Wand,et al.  Kernel Smoothing , 1995 .

[26]  Karen Spärck Jones.  A statistical interpretation of term specificity and its application in retrieval , 2021, J. Documentation.

[27]  Yale Song,et al.  One-Class Conditional Random Fields for Sequential Anomaly Detection , 2013, IJCAI.

[28]  Mark Schwabacher,et al.  Unsupervised Anomaly Detection for Liquid-Fueled Rocket Propulsion Health Monitoring , 2007, J. Aerosp. Comput. Inf. Commun..

[29]  D. Rajan.  Probability, Random Variables, and Stochastic Processes , 2017 .

[30]  C. D. Kemp,et al.  Density Estimation for Statistics and Data Analysis , 1987 .