Sparsity in Multivariate Extremes with Applications to Anomaly Detection

Capturing the dependence structure of multivariate extreme events is a major concern in many fields involving the management of risks stemming from multiple sources, e.g. portfolio monitoring, insurance, environmental risk management and anomaly detection. One convenient (non-parametric) characterization of extremal dependence in the framework of multivariate Extreme Value Theory (EVT) is the angular measure, which provides direct information about the probable 'directions' of extremes, that is, the relative contribution of each feature/coordinate of the 'largest' observations. Modeling the angular measure in high dimensional problems is a major challenge for the multivariate analysis of rare events. The present paper proposes a novel methodology aiming at exhibiting a sparsity pattern within the dependence structure of extremes. This is done by estimating the amount of mass spread by the angular measure on representative sets of directions, corresponding to specific sub-cones of $R^d_+$. This dimension reduction technique paves the way towards scaling up existing multivariate EVT methods. Beyond a non-asymptotic study providing a theoretical validity framework for our method, we propose as a direct application a –first– anomaly detection algorithm based on multivariate EVT. This algorithm builds a sparse 'normal profile' of extreme behaviours, to be confronted with new (possibly abnormal) extreme observations. Illustrative experimental results provide strong empirical evidence of the relevance of our approach.

[1]  David A. Clifton,et al.  Novelty Detection with Multivariate Extreme Value Statistics , 2011, J. Signal Process. Syst..

[2]  S. Coles,et al.  Modelling Extreme Multivariate Events , 1991 .

[3]  L. Haan,et al.  Modelling Multivariate Extremes , 1999 .

[4]  Holger Rootzén,et al.  Extreme Values in Finance, Telecommunications, and the Environment , 2003 .

[5]  J. Teugels,et al.  Tail Index Estimation, Pareto Quantile Plots, and Regression Diagnostics , 1996 .

[6]  Graham J. Williams,et al.  On-Line Unsupervised Outlier Detection Using Finite Mixtures with Discounting Learning Algorithms , 2000, KDD '00.

[7]  Sushil Jajodia,et al.  Applications of Data Mining in Computer Security , 2002, Advances in Information Security.

[8]  Anne Sabourin,et al.  Learning the dependence structure of rare events: a non-asymptotic study , 2015, COLT.

[9]  Jung-Min Park,et al.  An overview of anomaly detection techniques: Existing solutions and latest technological trends , 2007, Comput. Networks.

[10]  Victoria J. Hodge,et al.  A Survey of Outlier Detection Methodologies , 2004, Artificial Intelligence Review.

[11]  S. Resnick,et al.  Limit theory for multivariate sample extremes , 1977 .

[12]  A. SABOURIN,et al.  Bayesian Dirichlet mixture model for multivariate extremes: A re-parametrization , 2014, Comput. Stat. Data Anal..

[13]  Hans-Peter Kriegel,et al.  OPTICS-OF: Identifying Local Outliers , 1999, PKDD.

[14]  Sameer Singh,et al.  Novelty detection: a review - part 1: statistical approaches , 2003, Signal Process..

[15]  L. Haan,et al.  A moment estimator for the index of an extreme-value distribution , 1989 .

[16]  H. Drees,et al.  Best Attainable Rates of Convergence for Estimators of the Stable Tail Dependence Function , 1998 .

[17]  J. Hüsler,et al.  Statistical Analysis of Extreme Values with Applications to Insurance, Finance, Hydrology and Other Fields , 2007 .

[18]  Bernhard Schölkopf,et al.  Estimating the Support of a High-Dimensional Distribution , 2001, Neural Computation.

[19]  Johan Segers,et al.  An M-Estimator for Tail Dependence in Arbitrary Dimensions , 2011, 1112.0905.

[20]  Zhi-Hua Zhou,et al.  Isolation Forest , 2008, 2008 Eighth IEEE International Conference on Data Mining.

[21]  M. Shyu,et al.  A Novel Anomaly Detection Scheme Based on Principal Component Classifier , 2003 .

[22]  L. Haan,et al.  Extreme value theory , 2006 .

[23]  L. Tarassenko,et al.  Bayesian Extreme Value Statistics for Novelty Detection in Gas-Turbine Engines , 2008, 2008 IEEE Aerospace Conference.

[24]  B. M. Hill,et al.  A Simple General Approach to Inference About the Tail of a Distribution , 1975 .

[25]  L. Haan,et al.  Nonparametric estimation of the spectral measure of an extreme value distribution , 2001 .

[26]  S. Coles,et al.  An Introduction to Statistical Modeling of Extreme Values , 2001 .

[27]  J. Segers,et al.  Maximum Empirical Likelihood Estimation of the Spectral Measure of an Extreme Value Distribution , 2008, 0812.3485.

[28]  Salvatore J. Stolfo,et al.  A Geometric Framework for Unsupervised Anomaly Detection , 2002, Applications of Data Mining in Computer Security.

[29]  Daniel Cooley,et al.  The pairwise beta distribution: A flexible parametric multivariate model for extremes , 2010, J. Multivar. Anal..

[30]  Robert D. Nowak,et al.  Learning Minimum Volume Sets , 2005, J. Mach. Learn. Res..

[31]  A. Stephenson HIGH‐DIMENSIONAL PARAMETRIC MODELLING OF MULTIVARIATE EXTREME EVENTS , 2009 .

[32]  Deyuan Li,et al.  Weighted Approximations of Tail Copula Processes with Application to Testing the Multivariate Extreme Value Condition , 2004 .

[33]  J. Segers Asymptotics of empirical copula processes under non-restrictive smoothness assumptions , 2010, 1012.2133.

[34]  Jan Beirlant,et al.  Bias-corrected estimation of stable tail dependence function , 2016, J. Multivar. Anal..

[35]  Hyoungjoo Lee,et al.  On-line novelty detection using the Kalman filter and extreme value theory , 2008, 2008 19th International Conference on Pattern Recognition.

[36]  Vic Barnett,et al.  Outliers in Statistical Data , 1980 .

[37]  Richard Lippmann,et al.  Analysis and Results of the 1999 DARPA Off-Line Intrusion Detection Evaluation , 2000, Recent Advances in Intrusion Detection.

[38]  J. Tawn Modelling multivariate extreme value distributions , 1990 .

[39]  Philip S. Yu,et al.  Outlier detection for high dimensional data , 2001, SIGMOD '01.

[40]  Richard L. Smith Estimating tails of probability distributions , 1987 .

[41]  L. Haan,et al.  Bias correction in multivariate extremes , 2015, 1504.00490.

[42]  S. Roberts EXTREME VALUE STATISTICS FOR NOVELTY DETECTION IN BIOMEDICAL DATA PROCESSING , 2000 .

[43]  Nicole A. Lazar,et al.  Statistics of Extremes: Theory and Applications , 2005, Technometrics.

[44]  Ali A. Ghorbani,et al.  A detailed analysis of the KDD CUP 99 data set , 2009, 2009 IEEE Symposium on Computational Intelligence for Security and Defense Applications.

[45]  Eleazar Eskin,et al.  Anomaly Detection over Noisy Data using Learned Probability Distributions , 2000, ICML.

[46]  Anne Sabourin,et al.  Sparse Representation of Multivariate Extremes with Applications to Anomaly Ranking , 2016, AISTATS.

[47]  S. Roberts Novelty detection using extreme value statistics , 1999 .

[48]  Brahim Brahimi Statistics of Bivariate Extreme Values , 2014 .

[49]  S. Resnick Extreme Values, Regular Variation, and Point Processes , 1987 .

[50]  VARUN CHANDOLA,et al.  Anomaly detection: A survey , 2009, CSUR.

[51]  A. Stephenson Simulating Multivariate Extreme Value Distributions of Logistic Type , 2003 .

[52]  Magnus Almgren,et al.  Recent Advances in Intrusion Detection , 2004, Lecture Notes in Computer Science.

[53]  Stephen J. Roberts,et al.  Extreme value statistics for novelty detection in biomedical signal processing , 2000 .

[54]  Jun Li,et al.  Thresholding Events of Extreme in Simultaneous Monitoring of Multiple Risks , 2009 .

[55]  7K Qiyongcheng ALMOST SURE CONVERGENCE OF THE STABLE TAIL EMPIRICAL DEPENDENCE FUNCTION IN MULTIVARIATE EXTREME STATISTICS , 2005 .

[56]  J. Nolan,et al.  Models for Dependent Extremes Using Stable Mixtures , 2007, 0711.2345.