Near real-time intrusion alert aggregation using concept-based learning

Intrusion detection systems generate a large number of streaming alerts. It can be overwhelming for analysts to quickly and effectively find related alerts stemmed from correlated attack actions. What if fast arriving alerts could be automatically processed with no prior knowledge to find related actions in near real-time? The Concept Learning for Intrusion Event Aggregation in Realtime (CLEAR) system aims to learn and update an evolving set of temporal 'concepts,' each consisting of aggregates of related alerts that exhibit similar statistical arrival patterns. With no training data, the system constructs the concepts in near real-time from statistically similar alert aggregates. Tracked concepts are then applied to incoming alerts for fast and high-fidelity aggregation. The concepts learned by CLEAR are significantly more unique and invariant when compared to those learned by alternative drift detection methods. Furthermore, it provides insights for how specific individual, or co-occuring, alerts arrive with distinct and consistent temporal patterns.

[1]  Hervé Debar,et al.  Processing intrusion detection alert aggregates with time series modeling , 2009, Inf. Fusion.

[2]  Li Wan,et al.  Heterogeneous Ensemble for Feature Drifts in Data Streams , 2012, PAKDD.

[3]  Francisco Herrera,et al.  A unifying view on dataset shift in classification , 2012, Pattern Recognit..

[4]  Hervé Debar,et al.  Aggregation and Correlation of Intrusion-Detection Alerts , 2001, Recent Advances in Intrusion Detection.

[5]  Lize Gu,et al.  An Efficient Alert Aggregation Method Based on Conditional Rough Entropy and Knowledge Granularity , 2020, Entropy.

[6]  Shanchieh Jay Yang,et al.  Time series forecasting of cyber attack intensity , 2017, CISRC.

[7]  Chun-Hung Richard Lin,et al.  Intrusion detection system: A comprehensive review , 2013, J. Netw. Comput. Appl..

[8]  Gordon Werner,et al.  CAPTURE: Cyberattack Forecasting Using Non-Stationary Features with Time Lags , 2019, 2019 IEEE Conference on Communications and Network Security (CNS).

[9]  Shanchieh Jay Yang,et al.  ASSERT: attack synthesis and separation with entropy redistribution towards predictive cyber defense , 2019, Cybersecur..

[10]  Leandro L. Minku,et al.  FEDD: Feature Extraction for Explicit Concept Drift Detection in time series , 2016, 2016 International Joint Conference on Neural Networks (IJCNN).

[11]  Moises Sudit,et al.  Mission Impact Assessment for Cyber Warfare , 2015, Intelligent Methods for Cyber Warfare.

[12]  Heng Wang,et al.  Concept drift detection for streaming data , 2015, 2015 International Joint Conference on Neural Networks (IJCNN).

[13]  F. Massey.  The Kolmogorov-Smirnov Test for Goodness of Fit , 1951 .

[14]  Girijesh Prasad,et al.  EWMA model based shift-detection methods for detecting covariate shifts in non-stationary environments , 2015, Pattern Recognit..

[15]  S. W. Roberts,et al.  Control Chart Tests Based on Geometric Moving Averages , 2000, Technometrics.

[16]  Gregory Ditzler,et al.  Learning in Nonstationary Environments: A Survey , 2015, IEEE Computational Intelligence Magazine.

[17]  Milan Cermák,et al.  A graph-based representation of relations in network security alert sharing platforms , 2017, 2017 IFIP/IEEE Symposium on Integrated Network and Service Management (IM).

[18]  Jean Paul Barddal,et al.  Analyzing the Impact of Feature Drifts in Streaming Learning , 2015, ICONIP.

[19]  Jean Paul Barddal,et al.  A survey on feature drift adaptation: Definition, benchmark, challenges and future directions , 2017, J. Syst. Softw..

[20]  Gordon Werner,et al.  Leveraging Intra-Day Temporal Variations to Predict Daily Cyberattack Activity , 2018, 2018 IEEE International Conference on Intelligence and Security Informatics (ISI).

[21]  Christos Faloutsos,et al.  The self-feeding process: a unifying model for communication dynamics in the web , 2013, WWW.

[22]  Stefano Zanero,et al.  Reducing false positives in anomaly detectors through fuzzy alert aggregation , 2009, Inf. Fusion.

[23]  Ricard Gavaldà,et al.  Learning from Time-Changing Data with Adaptive Windowing , 2007, SDM.

[24]  A. Bifet,et al.  Early Drift Detection Method , 2005 .

[25]  Jan Vykopal,et al.  Exchanging security events: Which and how many alerts can we aggregate? , 2017, 2017 IFIP/IEEE Symposium on Integrated Network and Service Management (IM).