Behavior Flow Graph Construction from System Logs for Anomaly Analysis

Anomaly analysis plays a significant role in building a secure and reliable system. Raw system logs contain important system information, such as execution paths and execution time. People often use system logs for fault diagnosis and root cause localization. However, due to the complexity of raw system logs, these tasks can be arduous and ineffective. To solve this problem, we propose ETGC (Event Topology Graph Construction), a method for mining event topology graph of the normal execution status of systems. ETGC mines the dependency relationship between events and generates the event topology graph based on the maximum spanning tree. We evaluate the proposed method on data sets of real systems to demonstrate the effectiveness of our approach.

[1]  Xiao Yu,et al.  CloudSeer: Workflow Monitoring of Cloud Infrastructures via Interleaved Logs , 2016, ASPLOS.

[2]  Jennifer Neville,et al.  Structured Comparative Analysis of Systems Logs to Diagnose Performance Problems , 2012, NSDI.

[3]  Lin Yang,et al.  LOGAN: Problem Diagnosis in the Cloud Using Log-Based Reference Models , 2016, 2016 IEEE International Conference on Cloud Engineering (IC2E).

[4]  Ying Li,et al.  LogSed: Anomaly Diagnosis through Mining Time-Weighted Control Flow Graph in Logs , 2017, 2017 IEEE 10th International Conference on Cloud Computing (CLOUD).

[5]  Shilin He,et al.  Experience Report: System Log Analysis for Anomaly Detection , 2016, 2016 IEEE 27th International Symposium on Software Reliability Engineering (ISSRE).

[6]  Qiang Fu,et al.  Mining Invariants from Console Logs for System Problem Detection , 2010, USENIX Annual Technical Conference.

[7]  Michael I. Jordan,et al.  Detecting large-scale system problems by mining console logs , 2009, SOSP '09.

[8]  Yu Zhang,et al.  Log Clustering Based Problem Identification for Online Service Systems , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering Companion (ICSE-C).

[9]  Fernando Pereira,et al.  Non-Projective Dependency Parsing using Spanning Tree Algorithms , 2005, HLT.

[10]  Tao Li,et al.  Event Extraction from Streaming System Logs , 2018, ICISA.

[11]  Tao Li,et al.  Mining temporal patterns without predefined time windows , 2004, Fourth IEEE International Conference on Data Mining (ICDM'04).