Anomaly detection in dynamic networks: a survey

Anomaly detection is an important problem with multiple applications, and thus has been studied for decades in various research domains. In the past decade there has been a growing interest in anomaly detection in data represented as networks, or graphs, largely because of their robust expressiveness and their natural ability to represent complex relationships. Originally, techniques focused on anomaly detection in static graphs, which do not change and are capable of representing only a single snapshot of data. As real-world networks are constantly changing, there has been a shift in focus to dynamic graphs, which evolve over time.

[1]  D. Hand,et al.  Bayesian anomaly detection methods for social networks , 2010, 1011.1788.

[2]  Pang-Ning Tan,et al.  Detection and Characterization of Anomalies in Multivariate Time Series , 2009, SDM.

[3]  Ryan A. Rossi,et al.  Modeling dynamic behavior in large evolving graphs , 2013, WSDM.

[4]  Yingjie Zhou,et al.  Large-scale IP network behavior anomaly detection and identification using substructure-based approach and multivariate time series mining , 2012, Telecommun. Syst..

[5]  A.N. Srivastava,et al.  Enabling the discovery of recurring anomalies in aerospace problem reports using high-dimensional clustering techniques , 2006, 2006 IEEE Aerospace Conference.

[6]  Danai Koutra,et al.  DELTACON: A Principled Massive-Graph Similarity Function , 2013, SDM.

[7]  Philip S. Yu,et al.  GraphScope: parameter-free mining of large time-evolving graphs , 2007, KDD '07.

[8]  Daniel B. Neill,et al.  Non-parametric scan statistics for event detection and forecasting in heterogeneous social media graphs , 2014, KDD.

[9]  Tamara G. Kolda,et al.  Scalable Tensor Decompositions for Multi-aspect Data Mining , 2008, 2008 Eighth IEEE International Conference on Data Mining.

[10]  Diane J. Cook,et al.  Graph-based anomaly detection , 2003, KDD '03.

[11]  Jun Gao,et al.  Incremental Local Evolutionary Outlier Detection for Dynamic Social Networks , 2013, ECML/PKDD.

[12]  James Won-Ki Hong,et al.  Traffic dispersion graph based anomaly detection , 2011, SoICT.

[13]  Nagiza F. Samatova,et al.  Discovery of extreme events-related communities in contrasting groups of physical system networks , 2012, Data Mining and Knowledge Discovery.

[14]  Peter Grünwald,et al.  A tutorial introduction to the minimum description length principle , 2004, ArXiv.

[15]  Tina Eliassi-Rad,et al.  Detecting Novel Discrepancies in Communication Networks , 2010, 2010 IEEE International Conference on Data Mining.

[16]  Yizhou Sun,et al.  On community outliers and their efficient detection in information networks , 2010, KDD.

[17]  Charu C. Aggarwal,et al.  Event Detection in Social Streams , 2012, SDM.

[18]  P. Young,et al.  Time series analysis, forecasting and control , 1972, IEEE Transactions on Automatic Control.

[19]  Nagiza F. Samatova,et al.  Practical Graph Mining with R , 2013 .

[20]  J. Reichardt,et al.  Statistical mechanics of community detection. , 2006, Physical review. E, Statistical, nonlinear, and soft matter physics.

[21]  Venkatesh Saligrama,et al.  Video anomaly detection based on local statistical aggregates , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[22]  Danai Koutra,et al.  Graph based anomaly detection and description: a survey , 2014, Data Mining and Knowledge Discovery.

[23]  Danai Koutra,et al.  TensorSplat: Spotting Latent Anomalies in Time , 2012, 2012 16th Panhellenic Conference on Informatics.

[24]  Bernard Chazelle,et al.  The discrepancy method - randomness and complexity , 2000 .

[25]  Hans-Peter Kriegel,et al.  LOF: identifying density-based local outliers , 2000, SIGMOD '00.

[26]  Ambuj K. Singh,et al.  Mining Evolving Network Processes , 2013, 2013 IEEE 13th International Conference on Data Mining.

[27]  Lawrence B. Holder,et al.  Mining Graph Data: Cook/Mining Graph Data , 2006 .

[28]  Nitesh V. Chawla,et al.  Complex networks as a unified framework for descriptive analysis and predictive modeling in climate science , 2011, Stat. Anal. Data Min..

[29]  Tao Zhang,et al.  Anomalous path detection with hardware support , 2005, CASES '05.

[30]  Charu C. Aggarwal,et al.  Outlier Detection for Temporal Data , 2014, Outlier Detection for Temporal Data.

[31]  Nikos D. Sidiropoulos,et al.  ParCube: Sparse Parallelizable Tensor Decompositions , 2012, ECML/PKDD.

[32]  Yizhou Sun,et al.  Community Trend Outlier Detection Using Soft Temporal Pattern Mining , 2012, ECML/PKDD.

[33]  Huan Liu,et al.  Subspace clustering for high dimensional data: a review , 2004, SKDD.

[34]  Ambuj K. Singh,et al.  NetSpot: Spotting Significant Anomalous Regions on Dynamic Networks , 2013, SDM.

[35]  Jennifer Rexford,et al.  Sensitivity of PCA for traffic anomaly detection , 2007, SIGMETRICS '07.

[36]  Philip S. Yu,et al.  Colibri: fast mining of large static and dynamic graphs , 2008, KDD.

[37]  Jon Kleinberg,et al.  KDD '07: Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining , 2007, KDD 2007.

[38]  Tanya Y. Berger-Wolf,et al.  Constant-factor approximation algorithms for identifying dynamic communities , 2009, KDD.

[39]  Christos Faloutsos,et al.  Parallel crawling for online social networks , 2007, WWW '07.

[40]  Douglas M. Hawkins.  Identification of Outliers , 1980, Monographs on Applied Probability and Statistics.

[41]  Philip S. Yu,et al.  Outlier detection in graph streams , 2011, 2011 IEEE 27th International Conference on Data Engineering.

[42]  Vipin Kumar,et al.  Anomaly Detection for Discrete Sequences: A Survey , 2012, IEEE Transactions on Knowledge and Data Engineering.

[43]  Danai Koutra,et al.  Unifying Guilt-by-Association Approaches: Theorems and Fast Algorithms , 2011, ECML/PKDD.

[44]  Christos Faloutsos,et al.  LOCI: fast outlier detection using the local correlation integral , 2003, Proceedings 19th International Conference on Data Engineering (Cat. No.03CH37405).

[45]  Curtis B. Storlie,et al.  Scan Statistics for the Online Detection of Locally Anomalous Subgraphs , 2013, Technometrics.

[46]  Christophe Diot,et al.  Diagnosing network-wide traffic anomalies , 2004, SIGCOMM.

[47]  Kenji Yamanishi,et al.  Network anomaly detection based on Eigen equation compression , 2009, KDD.

[48]  Nagiza F. Samatova,et al.  Community-based anomaly detection in evolutionary networks , 2012, Journal of Intelligent Information Systems.

[49]  R. Balakrishnan,et al.  A textbook of graph theory , 1999 .

[50]  Ananthram Swami,et al.  Com2: Fast Automatic Discovery of Temporal ('Comet') Communities , 2014, PAKDD.

[51]  Brandon Pincombea,et al.  Anomaly Detection in Time Series of Graphs using ARMA Processes , 2007 .

[52]  J. Rissanen.  A UNIVERSAL PRIOR FOR INTEGERS AND ESTIMATION BY MINIMUM DESCRIPTION LENGTH , 1983 .

[53]  Jimeng Sun,et al.  Less is More: Compact Matrix Decomposition for Large Sparse Graphs , 2007, SDM.

[54]  Jimeng Sun,et al.  Beyond streams and graphs: dynamic tensor analysis , 2006, KDD '06.

[55]  Xuelong Li,et al.  A survey of graph edit distance , 2010, Pattern Analysis and Applications.

[56]  Pang-Ning Tan,et al.  A Robust Graph-Based Algorithm for Detection and Characterization of Anomalies in Noisy Multivariate Time Series , 2008, 2008 IEEE International Conference on Data Mining Workshops.

[57]  HolderLawrence,et al.  Anomaly detection in data represented as graphs , 2007 .

[58]  Qinbao Song,et al.  A Fast Clustering-Based Feature Subset Selection Algorithm for High-Dimensional Data , 2013, IEEE Transactions on Knowledge and Data Engineering.

[59]  L. Tucker,et al.  Some mathematical notes on three-mode factor analysis , 1966, Psychometrika.

[60]  D. West.  Introduction to Graph Theory , 1995 .

[61]  Tanya Y. Berger-Wolf,et al.  A framework for community identification in dynamic social networks , 2007, KDD '07.

[62]  Jiawei Han,et al.  Community Distribution Outlier Detection in Heterogeneous Information Networks , 2013, ECML/PKDD.

[63]  Arvind Ramanathan,et al.  An Online Approach for Mining Collective Behaviors from Molecular Dynamics Simulations , 2009, RECOMB.

[64]  Chrisil Arackaparambil,et al.  Wiki-Watchdog: Anomaly Detection in Wikipedia Through a Distributional Lens , 2011, 2011 IEEE/WIC/ACM International Conferences on Web Intelligence and Intelligent Agent Technology.

[65]  Yizhou Sun,et al.  Integrating community matching and outlier detection for mining evolutionary community outliers , 2012, KDD.

[66]  Srinivasan Parthasarathy,et al.  LOADED: link-based outlier and anomaly detection in evolving data sets , 2004, Fourth IEEE International Conference on Data Mining (ICDM'04).

[67]  Tina Eliassi-Rad,et al.  DAPA-V10: Discovery and Analysis of Patterns and Anomalies in Volatile Time-Evolving Networks , 2009 .

[68]  Moses Charikar,et al.  Similarity estimation techniques from rounding algorithms , 2002, STOC '02.

[69]  Santo Fortunato,et al.  Community detection in graphs , 2009, ArXiv.

[70]  Heng Wang,et al.  Locality Statistics for Anomaly Detection in Time Series of Graphs , 2013, IEEE Transactions on Signal Processing.

[71]  Andrea Lancichinetti,et al.  Community detection algorithms: a comparative analysis: invited presentation, extended abstract , 2009, VALUETOOLS.

[72]  Aleksandar Lazarevic,et al.  Incremental Local Outlier Detection for Data Streams , 2007, 2007 IEEE Symposium on Computational Intelligence and Data Mining.

[73]  Jeremy Kepner,et al.  A scalable signal processing architecture for massive graph analysis , 2012, 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[74]  Shoichiro Asano,et al.  Detecting Anomalous Traffic using Communication Graphs , 2011 .

[75]  Ben Y. Zhao,et al.  Measurement-calibrated graph models for social network experiments , 2010, WWW '10.

[76]  Clara Pizzuti,et al.  Outlier mining in large high-dimensional data sets , 2005, IEEE Transactions on Knowledge and Data Engineering.

[77]  Jing Xu,et al.  Intrusion Detection using Continuous Time Bayesian Networks , 2010, J. Artif. Intell. Res..

[78]  Charu C. Aggarwal,et al.  Outlier Analysis , 2013, Springer New York.

[79]  F. Chung,et al.  Spectra of random graphs with given expected degrees , 2003, Proceedings of the National Academy of Sciences of the United States of America.

[80]  J. Rissanen,et al.  Modeling By Shortest Data Description* , 1978, Autom..

[81]  Ian Davidson,et al.  On constrained spectral clustering and its applications , 2012, Data Mining and Knowledge Discovery.

[82]  Gregory M. Provan,et al.  Generating Application-Specific Benchmark Models for Complex Systems , 2008, AAAI.

[83]  Jiebo Luo,et al.  Video detection anomaly via low-rank and sparse decompositions , 2012, 2012 Western New York Image Processing Workshop.

[84]  Hongliang Fei,et al.  Anomaly localization for network data streams with graph joint sparse PCA , 2011, KDD.

[85]  Ambuj K. Singh,et al.  Mining Heavy Subgraphs in Time-Evolving Networks , 2011, 2011 IEEE 11th International Conference on Data Mining.

[86]  A. Stechow,et al.  Decomposition , 1902, The Indian medical gazette.

[87]  Daniel Dajun Zeng,et al.  A Link Prediction Approach to Anomalous Email Detection , 2006, 2006 IEEE International Conference on Systems, Man and Cybernetics.

[88]  Lawrence B. Holder,et al.  Anomaly detection in data represented as graphs , 2007, Intell. Data Anal..

[89]  David D. Jensen,et al.  The case for anomalous link discovery , 2005, SKDD.

[90]  Charu C. Aggarwal,et al.  On Anomalous Hotspot Discovery in Graph Streams , 2013, 2013 IEEE 13th International Conference on Data Mining.

[91]  Charu C. Aggarwal,et al.  Outlier Detection for Temporal Data: A Survey , 2014, IEEE Transactions on Knowledge and Data Engineering.

[92]  C. Faloutsos,et al.  EVENT DETECTION IN TIME SERIES OF MOBILE COMMUNICATION GRAPHS , 2010 .

[93]  Yumin Chen,et al.  Neighborhood outlier detection , 2010, Expert Syst. Appl..

[94]  David J. Marchette,et al.  Scan Statistics on Enron Graphs , 2005, Comput. Math. Organ. Theory.

[95]  Victoria J. Hodge,et al.  A Survey of Outlier Detection Methodologies , 2004, Artificial Intelligence Review.

[96]  Clara Pizzuti,et al.  Fast Outlier Detection in High Dimensional Spaces , 2002, PKDD.

[97]  Zhengding Lu,et al.  Community mining on dynamic weighted directed graphs , 2009, CIKM-CNIKM.

[98]  Benjamin A. Miller,et al.  Efficient anomaly detection in dynamic, attributed graphs: Emerging phenomena and big data , 2013, 2013 IEEE International Conference on Intelligence and Security Informatics.

[99]  Philip S. Yu,et al.  Local Correlation Tracking in Time Series , 2006, Sixth International Conference on Data Mining (ICDM'06).

[100]  Christos Faloutsos,et al.  oddball: Spotting Anomalies in Weighted Graphs , 2010, PAKDD.

[101]  Paul Barford,et al.  Intrusion as (anti)social communication: characterization and detection , 2012, KDD.

[102]  Priya Mahadevan,et al.  Systematic topology analysis and generation using degree correlations , 2006, SIGCOMM 2006.

[103]  VARUN CHANDOLA,et al.  Anomaly detection: A survey , 2009, CSUR.

[104]  Vic Barnett,et al.  Outliers in Statistical Data , 1980 .

[105]  Reda Alhajj,et al.  A comprehensive survey of numeric and symbolic outlier mining techniques , 2006, Intell. Data Anal..

[106]  Jorma Rissanen,et al.  Minimum Description Length Principle , 2010, Encyclopedia of Machine Learning.

[107]  Richard A. Harshman,et al.  Foundations of the PARAFAC procedure: Models and conditions for an "explanatory" multi-model factor analysis , 1970 .

[108]  Benjamin A. Miller,et al.  Eigenspace analysis for threat detection in social networks , 2011, 14th International Conference on Information Fusion.

[109]  Tanya Y. Berger-Wolf,et al.  Finding Communities in Dynamic Social Networks , 2011, 2011 IEEE 11th International Conference on Data Mining.

[110]  C. Bilgin.  Dynamic Network Evolution : Models , Clustering , Anomaly Detection , 2009 .

[111]  Philip S. Yu,et al.  Window-based Tensor Analysis on High-dimensional and Multi-aspect Streams , 2006, Sixth International Conference on Data Mining (ICDM'06).

[112]  Danai Koutra,et al.  NetSimile: A Scalable Approach to Size-Independent Network Similarity , 2012, ArXiv.

[113]  Miro Kraetzl,et al.  Using graph diameter for change detection in dynamic networks , 2006, Australas. J Comb..

[114]  Kwang-Ho Ro,et al.  Outlier detection for high-dimensional data , 2015 .

[115]  A. Laub,et al.  The singular value decomposition: Its computation and some applications , 1980 .

[116]  Vipin Kumar,et al.  Comparative Evaluation of Anomaly Detection Techniques for Sequence Data , 2008, 2008 Eighth IEEE International Conference on Data Mining.

[117]  Derek Greene,et al.  Tracking the Evolution of Communities in Dynamic Social Networks , 2010, 2010 International Conference on Advances in Social Networks Analysis and Mining.

[118]  Hector Garcia-Molina,et al.  Web graph similarity for anomaly detection , 2010, Journal of Internet Services and Applications.

[119]  Jae-Gil Lee,et al.  Temporal Outlier Detection in Vehicle Traffic Data , 2009, 2009 IEEE 25th International Conference on Data Engineering.

[120]  Lawrence B. Holder,et al.  Detecting Anomalies in Cargo Using Graph Properties , 2006, ISI.

[121]  S. V. Wiel,et al.  Graph Based Statistical Analysis of Network Traffic , 2011 .

[122]  Christos Faloutsos,et al.  Fast Robustness Estimation in Large Social Graphs: Communities and Anomaly Detection , 2012, SDM.

[123]  Tamara G. Kolda,et al.  Tensor Decompositions and Applications , 2009, SIAM Rev..

[124]  Lawrence B. Holder,et al.  Mining Graph Data , 2006 .

[125]  Nir Friedman,et al.  Probabilistic Graphical Models - Principles and Techniques , 2009 .

[126]  Johan A. K. Suykens,et al.  Least Squares Support Vector Machine Classifiers , 1999, Neural Processing Letters.

[127]  Ian T. Jolliffe,et al.  Principal Component Analysis , 2002, International Encyclopedia of Statistical Science.

[128]  Deepayan Chakrabarti,et al.  AutoPart: Parameter-Free Graph Partitioning and Outlier Detection , 2004, PKDD.

[129]  Hisashi Kashima,et al.  Eigenspace-based anomaly detection in computer systems , 2004, KDD.

[130]  Christos Faloutsos,et al.  TWave: High-order analysis of functional MRI , 2011, NeuroImage.

[131]  Steve Harenberg,et al.  Community detection in large‐scale networks: a survey and empirical evaluation , 2014 .

[132]  Christos Faloutsos,et al.  Netprobe: a fast and scalable system for fraud detection in online auction networks , 2007, WWW '07.

[133]  Hanghang Tong,et al.  Non-Negative Residual Matrix Factorization with Application to Graph Anomaly Detection , 2011, SDM.

[134]  Erhard Rahm,et al.  A survey of approaches to automatic schema matching , 2001, The VLDB Journal.

[135]  Gene H. Golub,et al.  Singular value decomposition and least squares solutions , 1970, Milestones in Matrix Computation.