Isconna: Streaming Anomaly Detection with Frequency and Patterns

An edge stream is a common form of presentation of dynamic networks. It can evolve with time, with new types of nodes or edges being continuously added. Existing methods for anomaly detection rely on edge occurrence counts or compare pattern snippets found in historical records. In this work, we propose Isconna, which focuses on both the frequency and the pattern of edge records. The burst detection component targets anomalies between individual timestamps, while the pattern detection component highlights anomalies across segments of timestamps. These two components together produce three intermediate scores, which are aggregated into the final anomaly score. Isconna does not actively explore or maintain pattern snippets; it instead measures the consecutive presence and absence of edge records. Isconna is an online algorithm, it does not keep the original information of edge records; only statistical values are maintained in a few count-min sketches (CMS). Isconna's space complexity $O(rc)$ is determined by two user-specific parameters, the size of CMSs. In worst case, Isconna's time complexity can be up to $O(rc)$, but it can be amortized in practice. Experiments show that Isconna outperforms five state-of-the-art frequency- and/or pattern-based baselines on six real-world datasets with up to 20 million edge records.

[1]  Toon Calders,et al.  Zips: mining compressing sequential patterns in streams , 2013, IDEA@KDD.

[2]  Yizhou Sun,et al.  Integrating community matching and outlier detection for mining evolutionary community outliers , 2012, KDD.

[3]  Christos Faloutsos,et al.  DenseAlert: Incremental Dense-Subtensor Detection in Tensor Streams , 2017, KDD.

[4]  Donato Malerba,et al.  A Sliding Window Algorithm for Relational Frequent Patterns Mining from Data Streams , 2009, Discovery Science.

[5]  Sudipto Guha,et al.  SpotLight: Detecting Anomalies in Streaming Graphs , 2018, KDD.

[6]  Jari Saramäki,et al.  Temporal motifs in time-dependent networks , 2011, ArXiv.

[7]  Danai Koutra,et al.  Graph based anomaly detection and description: a survey , 2014, Data Mining and Knowledge Discovery.

[8]  Jimeng Sun,et al.  Beyond streams and graphs: dynamic tensor analysis , 2006, KDD '06.

[9]  Venkatesan Guruswami,et al.  CopyCatch: stopping group attacks by spotting lockstep behavior in social networks , 2013, WWW.

[10]  Sutanay Choudhury,et al.  Frequent Subgraph Discovery in Large Attributed Streaming Graphs , 2014, BigMine.

[11]  Chao Lan,et al.  Anomaly Detection , 2018, Encyclopedia of GIS.

[12]  Christos Faloutsos,et al.  Patterns and anomalies in k-cores of real-world graphs with applications , 2018, Knowledge and Information Systems.

[13]  Christos Faloutsos,et al.  Fast Robustness Estimation in Large Social Graphs: Communities and Anomaly Detection , 2012, SDM.

[14]  Christos Faloutsos,et al.  MIDAS: Microcluster-Based Detector of Anomalies in Edge Streams , 2019, AAAI.

[15]  Austin R. Benson,et al.  Sampling Methods for Counting Temporal Motifs , 2019, WSDM.

[16]  Christos Faloutsos,et al.  oddball: Spotting Anomalies in Weighted Graphs , 2010, PAKDD.

[17]  Danai Koutra,et al.  Mining Persistent Activity in Continually Evolving Networks , 2020, KDD.

[18]  Balaraman Ravindran,et al.  COMMIT: A Scalable Approach to Mining Communication Motifs from Dynamic Networks , 2015, SIGMOD Conference.

[19]  Yang Zhang,et al.  Detecting Compromised Email Accounts from the Perspective of Graph Topology , 2016, CFI.

[20]  Nour Moustafa,et al.  UNSW-NB15: a comprehensive data set for network intrusion detection systems (UNSW-NB15 network data set) , 2015, 2015 Military Communications and Information Systems Conference (MilCIS).

[21]  Nikos Mamoulis,et al.  Flow Motifs in Interaction Networks , 2018, EDBT.

[22]  Deepayan Chakrabarti,et al.  AutoPart: Parameter-Free Graph Partitioning and Outlier Detection , 2004, PKDD.

[23]  Raghavendra Chalapathy University of Sydney,et al.  Deep Learning for Anomaly Detection: A Survey , 2019, ArXiv.

[24]  Ali A. Ghorbani,et al.  Developing Realistic Distributed Denial of Service (DDoS) Attack Dataset and Taxonomy , 2019, 2019 International Carnahan Conference on Security Technology (ICCST).

[25]  Michelangelo Ceci,et al.  Relational Frequent Patterns Mining for Novelty Detection from Data Streams , 2009, MLDM.

[26]  Hanghang Tong,et al.  Non-Negative Residual Matrix Factorization with Application to Graph Anomaly Detection , 2011, SDM.

[27]  Steve Harenberg,et al.  A Scalable Approach for Outlier Detection in Edge Streams Using Sketch-based Approximations , 2016, SDM.

[28]  Robert K. Cunningham,et al.  Results of the DARPA 1998 Offline Intrusion Detection Evaluation , 1999, Recent Advances in Intrusion Detection.

[29]  Panos Kalnis,et al.  Incremental Frequent Subgraph Mining on Large Evolving Graphs , 2017, IEEE Transactions on Knowledge and Data Engineering.

[30]  Ali A. Ghorbani,et al.  Toward developing a systematic approach to generate benchmark datasets for intrusion detection , 2012, Comput. Secur..

[31]  Aristides Gionis,et al.  Mining Frequent Patterns in Evolving Graphs , 2018, CIKM.

[32]  Markus Jakobsson,et al.  Social phishing , 2007, CACM.

[33]  Alejandro Zunino,et al.  An empirical comparison of botnet detection methods , 2014, Comput. Secur..

[34]  Chedy Raïssi,et al.  Mining Multidimensional Sequential Patterns over Data Streams , 2008, DaWaK.

[35]  Kumar Sricharan,et al.  Localizing anomalous changes in time-evolving graphs , 2014, SIGMOD Conference.

[36]  Hyun Ah Song,et al.  Graph-Based Fraud Detection in the Face of Camouflage , 2017, ACM Trans. Knowl. Discov. Data.

[37]  Charu C. Aggarwal,et al.  On Anomalous Hotspot Discovery in Graph Streams , 2013, 2013 IEEE 13th International Conference on Data Mining.

[38]  Nirvana Meratnia,et al.  Recognition of Periodic Behavioral Patterns from Streaming Mobility Data , 2013, MobiQuitous.

[39]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[40]  Rajeev Motwani,et al.  The PageRank Citation Ranking : Bringing Order to the Web , 1999, WWW 1999.

[41]  VARUN CHANDOLA,et al.  Anomaly detection: A survey , 2009, CSUR.

[42]  Rok Sosic,et al.  F-FADE: Frequency Factorization for Anomaly Detection in Edge Streams , 2020, WSDM.

[43]  Ali A. Ghorbani,et al.  Toward Generating a New Intrusion Detection Dataset and Intrusion Traffic Characterization , 2018, ICISSP.

[44]  Jure Leskovec,et al.  Motifs in Temporal Networks , 2016, WSDM.

[45]  Florent Masseglia,et al.  Mining sequential patterns from data streams: a centroid approach , 2006, Journal of Intelligent Information Systems.

[46]  Christos Faloutsos,et al.  Fast and Accurate Anomaly Detection in Dynamic Graphs with a Two-Pronged Approach , 2019, KDD.

[47]  Philip S. Yu,et al.  On dense pattern mining in graph streams , 2010, Proc. VLDB Endow..

[48]  Maguelonne Teisseire,et al.  Need For Speed : Mining Sequential Patterns in Data Streams , 2005, BDA.

[49]  George Karypis,et al.  Algorithms for Mining the Coevolving Relational Motifs in Dynamic Networks , 2015, ACM Trans. Knowl. Discov. Data.

[50]  Qi He,et al.  Communication motifs: a tool to characterize social communications , 2010, CIKM.

[51]  Panos Kalnis,et al.  GRAMI: Frequent Subgraph and Pattern Mining in a Single Large Graph , 2014, Proc. VLDB Endow..

[52]  Graham Cormode,et al.  An Improved Data Stream Summary: The Count-Min Sketch and Its Applications , 2004, LATIN.

[53]  Graham Cormode,et al.  An improved data stream summary: the count-min sketch and its applications , 2004, J. Algorithms.

[54]  Alexandre Termier,et al.  Anomaly Detection in Streams with Extreme Value Theory , 2017, KDD.

[55]  Christos Faloutsos,et al.  SedanSpot: Detecting Anomalies in Edge Streams , 2018, 2018 IEEE International Conference on Data Mining (ICDM).

[56]  Danai Koutra,et al.  TimeCrunch: Interpretable Dynamic Graph Summarization , 2015, KDD.

[57]  FaloutsosChristos,et al.  Catching Synchronized Behaviors in Large Networks , 2016 .