Universal Data Anomaly Detection via Inverse Generative Adversary Network

The problem of detecting data anomaly under unknown probability distributions is considered. Whereas the probability distribution of the anomaly-free data is unknown, anomaly-free training samples are assumed to be available. For anomaly data, neither the underlying probability distribution is known nor anomaly data samples are available. A deep learning approach coupled with a statistical test based on coincidence is proposed where an inverse generative adversary network is trained to transform data to the classical uniform vs. nonuniform hypothesis testing problem. The proposed approach is particularly effective to detect persistent anomalies whose distributions have an overlapping domain with that of the anomaly-free distribution.

[1]  Bo Zong,et al.  Deep Autoencoding Gaussian Mixture Model for Unsupervised Anomaly Detection , 2018, ICLR.

[2]  Mario Paolone,et al.  Real-time state estimation of the EPFL-campus medium-voltage grid by using PMUs , 2015, 2015 IEEE Power & Energy Society Innovative Smart Grid Technologies Conference (ISGT).

[3]  Bo Li,et al.  A deep learning enabled subspace spectral ensemble clustering approach for web anomaly detection , 2017, 2017 International Joint Conference on Neural Networks (IJCNN).

[4]  Two combinatorial tests of whether a sample has come from a given population. , 1950, Biometrika.

[5]  Edward C. van der Meulen,et al.  Entropy-Based Tests of Uniformity , 1981 .

[6]  Lang Tong,et al.  Learning the Unobservable: High-Resolution State Estimation via Deep Learning , 2019, 2019 57th Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[7]  Yoshua Bengio,et al.  Generative Adversarial Networks , 2014, ArXiv.

[8]  Aaron C. Courville,et al.  Improved Training of Wasserstein GANs , 2017, NIPS.

[9]  Sean P. Meyn,et al.  Generalized Error Exponents for Small Sample Universal Hypothesis Testing , 2012, IEEE Transactions on Information Theory.

[10]  M. F. Augusteijn,et al.  Neural network classification and novelty detection , 2002 .

[11]  Two combinatorial tests of whether a sample has come from a given population. , 1950 .

[12]  Hans-Peter Kriegel,et al.  LOF: identifying density-based local outliers , 2000, SIGMOD '00.

[13]  Alon Orlitsky,et al.  A Competitive Test for Uniformity of Monotone Distributions , 2013, AISTATS.

[14]  Liam Paninski,et al.  A Coincidence-Based Test for Uniformity Given Very Sparsely Sampled Discrete Data , 2008, IEEE Transactions on Information Theory.

[15]  Bernhard Schölkopf,et al.  Support Vector Method for Novelty Detection , 1999, NIPS.

[16]  Artur Czumaj,et al.  Testing monotone continuous distributions on high-dimensional real cubes , 2010, SODA '10.

[17]  Randy C. Paffenroth,et al.  Anomaly Detection with Robust Deep Autoencoders , 2017, KDD.

[18]  David A. Clifton,et al.  A review of novelty detection , 2014, Signal Process..

[19]  Lang Tong,et al.  Malicious Data Attacks on the Smart Grid , 2011, IEEE Transactions on Smart Grid.

[20]  Mauro Conti,et al.  A Survey of Man In The Middle Attacks , 2016, IEEE Communications Surveys & Tutorials.

[21]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[22]  VARUN CHANDOLA,et al.  Anomaly detection: A survey , 2009, CSUR.

[23]  Léon Bottou,et al.  Wasserstein GAN , 2017, ArXiv.

[24]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[25]  Yu Cheng,et al.  Deep Structured Energy Based Models for Anomaly Detection , 2016, ICML.

[26]  Georg Langs,et al.  f‐AnoGAN: Fast unsupervised anomaly detection with generative adversarial networks , 2019, Medical Image Anal..