Anomaly Detection of Time Series With Smoothness-Inducing Sequential Variational Auto-Encoder

Deep generative models have demonstrated their effectiveness in learning latent representation and modeling complex dependencies of time series. In this article, we present a smoothness-inducing sequential variational auto-encoder (VAE) (SISVAE) model for the robust estimation and anomaly detection of multidimensional time series. Our model is based on VAE, and its backbone is fulfilled by a recurrent neural network to capture latent temporal structures of time series for both the generative model and the inference model. Specifically, our model parameterizes mean and variance for each time-stamp with flexible neural networks, resulting in a nonstationary model that can work without the assumption of constant noise as commonly made by existing Markov models. However, such flexibility may cause the model fragile to anomalies. To achieve robust density estimation which can also benefit detection tasks, we propose a smoothness-inducing prior over possible estimations. The proposed prior works as a regularizer that places penalty at nonsmooth reconstructions. Our model is learned efficiently with a novel stochastic gradient variational Bayes estimator. In particular, we study two decision criteria for anomaly detection: reconstruction probability and reconstruction error. We show the effectiveness of our model on both synthetic data sets and public real-world benchmarks.

[1]  Christian Osendorfer,et al.  Learning Stochastic Recurrent Networks , 2014, NIPS 2014.

[2]  Han Zou,et al.  Non-Parametric Outliers Detection in Multiple Time Series A Case Study: Power Grid Data Analysis , 2018, AAAI.

[3]  David J. Hill,et al.  Anomaly detection in streaming environmental sensor data: A data-driven modeling approach , 2010, Environ. Model. Softw..

[4]  Christopher Burgess,et al.  beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework , 2016, ICLR 2016.

[5]  Freda Kemp,et al.  An Introduction to Sequential Monte Carlo Methods , 2003 .

[6]  Yoshua Bengio,et al.  A Recurrent Latent Variable Model for Sequential Data , 2015, NIPS.

[7]  VARUN CHANDOLA,et al.  Anomaly detection: A survey , 2009, CSUR.

[8]  Matthias Durr,et al.  Smoothness Priors Analysis Of Time Series , 2016 .

[9]  Diederik P. Kingma,et al.  Stochastic Gradient VB and the Variational Auto-Encoder , 2013 .

[10]  Gilles Clermont,et al.  Outlier detection for patient monitoring and alerting , 2013, J. Biomed. Informatics.

[11]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[12]  W. Enders Applied Econometric Time Series , 1994 .

[13]  Nando de Freitas,et al.  An Introduction to Sequential Monte Carlo Methods , 2001, Sequential Monte Carlo Methods in Practice.

[14]  Vipin Kumar,et al.  Anomaly Detection for Discrete Sequences: A Survey , 2012, IEEE Transactions on Knowledge and Data Engineering.

[15]  Justin Bayer,et al.  Variational Inference for On-line Anomaly Detection in High-Dimensional Time Series , 2016, ArXiv.

[16]  L. Glass,et al.  Oscillation and chaos in physiological control systems. , 1977, Science.

[17]  Sungzoon Cho,et al.  Variational Autoencoder based Anomaly Detection using Reconstruction Probability , 2015 .

[18]  Marius Kloft,et al.  Hidden Markov Anomaly Detection , 2015, ICML.

[19]  Razvan Pascanu,et al.  On the difficulty of training recurrent neural networks , 2012, ICML.

[20]  Christos Faloutsos,et al.  DynaMMo: mining and summarization of coevolving sequences with missing values , 2009, KDD.

[21]  Sung-Bae Cho,et al.  Zero-day malware detection using transferred generative adversarial networks based on deep autoencoders , 2018, Inf. Sci..

[22]  Christopher Leckie,et al.  High-dimensional and large-scale anomaly detection using a linear one-class SVM with deep learning , 2016, Pattern Recognit..

[23]  Jun S. Liu,et al.  Sequential Monte Carlo methods for dynamic systems , 1997 .

[24]  Charu C. Aggarwal,et al.  Outlier Detection for Temporal Data: A Survey , 2014, IEEE Transactions on Knowledge and Data Engineering.

[25]  Paul J. Werbos,et al.  Backpropagation Through Time: What It Does and How to Do It , 1990, Proc. IEEE.

[26]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[27]  R. Tsay,et al.  Outlier Detection in Multivariate Time Series by Projection Pursuit , 2006 .

[28]  Yang Feng,et al.  Unsupervised Anomaly Detection via Variational Auto-Encoder for Seasonal KPIs in Web Applications , 2018, WWW.

[29]  Johan A. K. Suykens,et al.  Enhancing Dynamic Soft Sensors based on DPLS: a Temporal Smoothness Regularization Approach , 2015 .

[30]  Göran Falkman,et al.  Anomaly detection in sea traffic - A comparison of the Gaussian Mixture Model and the Kernel Density Estimator , 2009, 2009 12th International Conference on Information Fusion.

[31]  Felipe Albertao,et al.  Incremental dictionary learning for fault detection with applications to oil pipeline leakage detection , 2011 .

[32]  Anna Liao,et al.  Open μPMU: A real world reference distribution micro-phasor measurement unit data set for research and application development: , 2016 .

[33]  J. A. Stewart,et al.  Nonlinear Time Series Analysis , 2015 .

[34]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.

[35]  Takaya Saito,et al.  The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets , 2015, PloS one.

[36]  Mark Goadrich,et al.  The relationship between Precision-Recall and ROC curves , 2006, ICML.

[37]  Simon J. Godsill,et al.  On sequential Monte Carlo sampling methods for Bayesian filtering , 2000, Stat. Comput..

[38]  Bernhard Schölkopf,et al.  Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations , 2018, ICML.

[39]  Hanghang Tong,et al.  Facets: Fast Comprehensive Mining of Coevolving High-order Time Series , 2015, KDD.

[40]  Alexandre Termier,et al.  Anomaly Detection in Streams with Extreme Value Theory , 2017, KDD.

[41]  Jeffrey L. Elman,et al.  Finding Structure in Time , 1990, Cogn. Sci..

[42]  Martin Meckesheimer,et al.  Automatic outlier detection for time series: an application to sensor data , 2007, Knowledge and Information Systems.

[43]  Jae-Gil Lee,et al.  Temporal Outlier Detection in Vehicle Traffic Data , 2009, 2009 IEEE 25th International Conference on Data Engineering.

[44]  Ole Winther,et al.  Sequential Neural Models with Stochastic Layers , 2016, NIPS.

[45]  R. Tsay,et al.  Outliers in multivariate time series , 2000 .

[46]  Xi Chen,et al.  Direct Robust Matrix Factorizatoin for Anomaly Detection , 2011, 2011 IEEE 11th International Conference on Data Mining.

[47]  Jonathan D. Cryer,et al.  Time Series Analysis , 1986 .

[48]  Randy C. Paffenroth,et al.  Anomaly Detection with Robust Deep Autoencoders , 2017, KDD.

[49]  Hanghang Tong,et al.  Fast Mining of a Network of Coevolving Time Series , 2015, SDM.

[50]  P. Burridge,et al.  Additive Outlier Detection Via Extreme‐Value Theory , 2006 .

[51]  Mooi Choo Chuah,et al.  ECG Anomaly Detection via Time Series Analysis , 2007, ISPA Workshops.

[52]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[53]  Stefano Ermon,et al.  Bias and Generalization in Deep Generative Models: An Empirical Study , 2018, NeurIPS.

[54]  Charu C. Aggarwal,et al.  Outlier Analysis , 2013, Springer New York.