An unsupervised framework for detecting anomalous messages from syslog log files

System logs provide valuable information about the health status of IT systems and computer networks. Therefore, log file monitoring has been identified as an important system and network management technique. While many solutions have been developed for monitoring known log messages, the detection of previously unknown error conditions has remained a difficult problem. In this paper, we present a novel data mining based framework for detecting anomalous log messages from syslog- based system log files. We also describe the implementation and performance of the framework in a large organizational network.

[1]  Akio Watanabe,et al.  Proactive failure detection learning generation patterns of large-scale network logs , 2015, Conference on Network and Service Management.

[2]  Risto Vaarandi,et al.  Bbuzz: A bit-aware fuzzing framework for network protocol systematic reverse engineering and analysis , 2017, MILCOM 2017 - 2017 IEEE Military Communications Conference (MILCOM).

[3]  Michael I. Jordan,et al.  Detecting large-scale system problems by mining console logs , 2009, SOSP '09.

[4]  Errin W. Fulp,et al.  Using Syslog Message Sequences for Predicting Disk Failures , 2010, LISA.

[5]  Thomas Reidemeister,et al.  Mining unstructured log files for recurrent fault diagnosis , 2011, 12th IFIP/IEEE International Symposium on Integrated Network Management (IM 2011) and Workshops.

[6]  Risto Vaarandi,et al.  A data clustering algorithm for mining patterns from event logs , 2003, Proceedings of the 3rd IEEE Workshop on IP Operations & Management (IPOM 2003) (IEEE Cat. No.03EX764).

[7]  Alexander Aiken,et al.  Alert Detection in System Logs , 2008, 2008 Eighth IEEE International Conference on Data Mining.

[8]  Risto Vaarandi,et al.  Simple event correlator - Best practices for creating scalable configurations , 2015, 2015 IEEE International Multi-Disciplinary Conference on Cognitive Methods in Situation Awareness and Decision.

[9]  Risto Vaarandi,et al.  LogCluster - A data clustering and pattern mining algorithm for event logs , 2015, 2015 11th International Conference on Network and Service Management (CNSM).

[10]  Kenji Yamanishi,et al.  Dynamic syslog mining for network failure monitoring , 2005, KDD '05.

[11]  Evangelos E. Milios,et al.  Spatio-temporal decomposition, clustering and identification for alert detection in system logs , 2012, SAC '12.

[12]  Adetokunbo Makanju,et al.  Exploring Event Log Analysis with Minimum Apriori Information , 2012 .

[13]  B. Jiang.  Head/Tail Breaks: A New Classification Scheme for Data with a Heavy-Tailed Distribution , 2012, 1209.2801.

[14]  Feifei Li,et al.  DeepLog: Anomaly Detection and Diagnosis from System Logs through Deep Learning , 2017, CCS.

[15]  Stephen E. Hansen,et al.  Automated System Monitoring and Notification with Swatch , 1993, LISA.

[16]  Shilin He,et al.  Experience Report: System Log Analysis for Anomaly Detection , 2016, 2016 IEEE 27th International Symposium on Software Reliability Engineering (ISSRE).

[17]  Felix Salfner,et al.  Event-based Failure Prediction: An Extended Hidden Markov Model Approach , 2008, Ausgezeichnete Informatikdissertationen.

[18]  Chris Lonvick,et al.  The BSD Syslog Protocol , 2001, RFC.