相关论文
Dynamic syslog mining for network failure monitoring
Abstract:Syslog monitoring technologies have recently received vast attentions in the areas of network management and network monitoring. They are used to address a wide range of important issues including network failure symptom detection and event correlation discovery. Syslogs are intrinsically dynamic in the sense that they form a time series and that their behavior may change over time. This paper proposes a new methodology of dynamic syslog mining in order to detect failure symptoms with higher confidence and to discover sequential alarm patterns among computer devices. The key ideas of dynamic syslog mining are 1) to represent syslog behavior using a mixture of Hidden Markov Models, 2) to adaptively learn the model using an on-line discounting learning algorithm in combination with dynamic selection of the optimal number of mixture components, and 3) to give anomaly scores using universal test statistics with a dynamically optimized threshold. Using real syslog data we demonstrate the validity of our methodology in the scenarios of failure symptom detection, emerging pattern identification, and correlation discovery.
相关论文分享
暂无分享，去 创建一个
参考文献
[1] Andrew J. Viterbi,et al. Error bounds for convolutional codes and an asymptotically optimum decoding algorithm , 1967, IEEE Trans. Inf. Theory.
[2] L. Baum,et al. A Maximization Technique Occurring in the Statistical Analysis of Probabilistic Functions of Markov Chains , 1970 .
[3] Abraham Lempel,et al. Compression of individual sequences via variable-rate coding , 1978, IEEE Trans. Inf. Theory.
[4] Raphail E. Krichevsky,et al. The performance of universal encoding , 1981, IEEE Trans. Inf. Theory.
[5] Jorma Rissanen,et al. Universal coding, information, prediction, and estimation , 1984, IEEE Trans. Inf. Theory.
[6] Jacob Ziv,et al. On classification with empirically observed statistics and universal data compression , 1988, IEEE Trans. Inf. Theory.
[7] G. Jakobson,et al. Alarm correlation , 1993, IEEE Network.
[8] Stephen E. Hansen,et al. Automated System Monitoring and Notification with Swatch , 1993, LISA.
[9] Padhraic Smyth,et al. Markov monitoring with unknown states , 1994, IEEE J. Sel. Areas Commun..
[10] Ramakrishnan Srikant,et al. Mining sequential patterns , 1995, Proceedings of the Eleventh International Conference on Data Engineering.
[11] D. Ohsie,et al. High speed and robust event correlation , 1996, IEEE Commun. Mag..
[12] Geoffrey E. Hinton,et al. A View of the Em Algorithm that Justifies Incremental, Sparse, and other Variants , 1998, Learning in Graphical Models.
[13] Graham J. Williams,et al. On-line unsupervised outlier detection using finite mixtures with discounting learning algorithms , 2000, KDD '00.
[14] Malgorzata Steinder,et al. The present and future of event correlation: A need for end-to-end service fault localization , 2001 .
[15] Chris Lonvick,et al. The BSD Syslog Protocol , 2001, RFC.
[16] Kenji Yamanishi,et al. A unifying framework for detecting outliers and change points from non-stationary time series data , 2002, KDD.
[17] Risto Vaarandi,et al. SEC - a lightweight event correlation tool , 2002, IEEE Workshop on IP Operations and Management.
[18] Risto Vaarandi,et al. A data clustering algorithm for mining patterns from event logs , 2003, Proceedings of the 3rd IEEE Workshop on IP Operations & Management (IPOM 2003) (IEEE Cat. No.03EX764).
[19] Joseph L. Hellerstein,et al. Data-driven validation, completion and construction of event relationship networks , 2003, KDD '03.
[20] Kenji Yamanishi,et al. Dynamic model selection with its applications to computer security , 2004, Information Theory Workshop.
[21] Heikki Mannila,et al. Discovery of Frequent Episodes in Event Sequences , 1997, Data Mining and Knowledge Discovery.
[22] Heikki Mannila,et al. Rule Discovery in Telecommunication Alarm Data , 1999, Journal of Network and Systems Management.
[23] Malgorzata Steinder,et al. Probabilistic fault localization in communication systems using belief networks , 2004, IEEE/ACM Transactions on Networking.
[24] Graham J. Williams,et al. On-Line Unsupervised Outlier Detection Using Finite Mixtures with Discounting Learning Algorithms , 2000, KDD '00.
A Failure Detection and Prediction Mechanism for Enhancing Dependability of Data Centers
2012
摘要
Abstract—Modern data centers continue to grow in their scale and complexity. They are changing dynamically as well due to the addition and removal of system components, changing execution environments, frequent updates and upgrades, online repairs and more. Classical reliability theory and conventional methods do rarely consider the actual state of a system and are therefore not capable to reflect the dynamics of runtime systems and failure processes. In this paper, we present an unsupervised failure detection and prediction method using an ensemble of Bayesian models. It characterizes normal execution states of the system and detects anomalous behaviors. We implement a prototype of our failure detection and prediction mechanism and evaluate its performance on a data center test platform. Experimental results show that our proposed method can forecast failure dynamics with high accuracy.
An end-to-end data transformation process for increasing the information yield of system traces
2013
摘要
• A submitted manuscript is the author's version of the article upon submission and before peer-review. There can be important differences between the submitted version and the official published version of record. People interested in the research are advised to contact the author for the final version of the publication, or visit the DOI to the publisher's website. • The final author version and the galley proof are versions of the publication after peer review. • The final published version features the final layout of the paper including the volume, issue and page numbers.
Proactive failure detection learning generation patterns of large-scale network logs
2015 11th International Conference on Network and Service Management (CNSM)
2015
摘要
With the growth of services in IP networks, network operators are required to perform proactive operation that quickly detects the signs of critical failures and prevents future problems. Network log data, including router syslog, are rich sources for such operations. However, it has become impossible to find genuinely important logs that lead to serious problems due to the large volume and complexity of log data. We propose a log analysis system for proactive detection of failures. Our key observation is that the abnormality of logs depends on not just the keywords in the messages (e.g. ERROR, FAIL), but generation patterns such as burstiness. Our system consists of three functions: (i) extracting log templates automatically and quickly from a massive amount of unstructured log data; (ii) constructing log feature vectors to characterize the generation patterns of logs; and (iii) using a supervised machine learning approach to associate failures with the log data that appeared before them. We validated our system using real log data collected from a large network and determined its effectiveness.
Performance Metric Selection for Autonomic Anomaly Detection on Cloud Computing Systems
2011 IEEE Global Telecommunications Conference - GLOBECOM 2011
2011
摘要
With ever-growing complexity and dynamicity of cloud computing systems, dependability assurance has become a major concern in system design and management. In this paper, we propose a framework for autonomic anomaly detection in the cloud. Mutual information is exploited to quantify the relevance and redundancy among the large number of performance metrics. An incremental search algorithm is presented for metric selection. We apply principal component analysis to further reduce the metric dimension, while keeping the variance in the health- related data as much as possible. A detection mechanism with semi-supervised decision tree classifiers works on the reduce metric dimensionality and identifies anomalies. We have implemented a prototype of our autonomic anomaly detection framework and evaluated its performance on an institute-wide cloud computing system.
Modern MDL meets Data Mining Insights, Theory, and Practice
KDD
2019
摘要
When considering a data set it is often unknown how complex it is, and hence it is difficult to assess how rich a model for the data should be. Often these choices are swept under the carpet, ignored, left to the domain expert, but in practice this is highly unsatisfactory; domain experts do not know how to set k, what prior to choose, or how many degrees of freedom is optimal any more than we do. The Minimum Description Length (MDL) principle can answer the model selection problem from an intuitively appealing and clear viewpoint of information theory and data compression. In a nutshell, it asserts that the best model is the one that best compresses both the data and that model. It does not only imply the best strategy for model selection, but also gives a unifying viewpoint of designing optimal data mining algorithms for a wide range of issues, and has been very successfully applied to a wide range of data mining tasks, ranging from pattern mining, clustering, classification, text mining, graph mining, anomaly detection, up to causal inference. In this tutorial we do not only give an introduction to the basics of model selection, show important properties of MDL-based modelling, successful examples as well as pitfalls for how to apply MDL to solve data mining problems, but also introduce advanced topics on important new concepts in modern MDL (e.g, normalized maximum likelihood (NML), sequential NML, decomposed NML, and MDL change statistics) and emerging applications in dynamic settings.
Alert Detection in System Logs
2008 Eighth IEEE International Conference on Data Mining
2008
摘要
We present Nodeinfo, an unsupervised algorithm for anomaly detection in system logs. We demonstrate Nodeinfo's effectiveness on data from four of the world's most powerful supercomputers: using logs representing over 746 million processor-hours, in which anomalous events called alerts were manually tagged for scoring, we aim to automatically identify the regions of the log containing those alerts. We formalize the alert detection task in these terms, describe how Nodeinfo uses the information entropy of message terms to identify alerts, and present an online version of this algorithm, which is now in production use. This is the first work to investigate alert detection on (several) publicly-available supercomputer system logs, thereby providing a reproducible performance baseline.
GAUL: Gestalt Analysis of Unstructured Logs for Diagnosing Recurring Problems in Large Enterprise Storage Systems
2010 29th IEEE Symposium on Reliable Distributed Systems
2010
摘要
We present GAUL, a system to automate the whole log comparison between a new problem and the ones diagnosed in the past to identify recurring problems. GAUL uses a fuzzy match algorithm based on the contextual overlap between log lines and efficiently implements this using scalable index/search. The accuracy and efficiency of the comparison is further improved by leveraging problem set information and noise tolerance techniques. We evaluate GAUL using 4339 customer problems that occurred in all field deployments of an enterprise storage system over the course of a year. Our results show that with human-filtered logs, GAUL can identify the correct problem set 66% of the time among the top10 matches, which is 15% more accurate than the VSM system that uses cosine similarity and 19% more accurate than the ERRCMP system that uses error codes for log comparison. With unfiltered logs, the top10 match accuracy of GAUL is 40%, which is 22% more accurate than VSM and 26% more accurate than ERRCMP.
Stage-aware anomaly detection through tracking log points
Middleware
2014
摘要
We introduce Stage-aware Anomaly Detection (SAAD), a low-overhead real-time solution for detecting runtime anomalies in storage systems. Modern storage server architectures are multi-threaded and structured as a set of modules, which we call stages. SAAD leverages this to collect stage-level log summaries at runtime and to perform statistical analysis across stage instances. Stages that generate rare execution flows and/or register unusually high duration for regular flows at run-time indicate anomalies. SAAD makes two key contributions: i) limits the search space for root causes, by pinpointing specific anomalous code stages, and ii) reduces compute and storage requirements for log analysis, while preserving accuracy, through a novel technique based on log summarization. We evaluate SAAD on three distributed storage systems: HBase, Hadoop Distributed File System (HDFS), and Cassandra. We show that, with practically zero overhead, we uncover various anomalies in real-time.
Online System Problem Detection by Mining Patterns of Console Logs
2009 Ninth IEEE International Conference on Data Mining
2009
摘要
We describe a novel application of using data mining and statistical learning methods to automatically monitor and detect abnormal execution traces from console logs in an online setting. Different from existing solutions, we use a two stage detection system. The first stage uses frequent pattern mining and distribution estimation techniques to capture the dominant patterns (both frequent sequences and time duration). The second stage use principal component analysis based anomaly detection technique to identify actual problems. Using real system data from a 203-node Hadoop [1] cluster, we show that we can not only achieve highly accurate and fast problem detection, but also help operators better understand execution patterns in their system.
Sequential network change detection with its applications to ad impact relation analysis
2012 IEEE 12th International Conference on Data Mining
2012
摘要
We are concerned with the issue of tracking changes of variable dependencies from multivariate time series. Conventionally, this issue has been addressed in the batch scenario where the whole data set is given at once, and the change detection must be done in a retrospective way. This paper addresses this issue in a sequential scenario where multivariate data are sequentially input and the detection must be done in a sequential fashion. We propose a new method for sequential tracking of variable dependencies. In it we employ a Bayesian network as a representation of variable dependencies. The key ideas of our method are: (1) we extend the theory of dynamic model selection, which has been developed in the batch-learning scenario, into the sequential setting, and apply it to our issue, (2) we conduct the change detection sequentially using dynamic programming per a window where we employ the Hoeffding’s bound to automatically determine the window size. We empirically demonstrate that our proposed method is able to perform change detection more efficiently than a conventional batch method. Further, we give a new framework of an application of variable dependency change detection, which we call Ad Impact Relation analysis (AIR). In it, we detect the time point when a commercial message advertisement has given an impact on the market and effectively visualize the impact through network changes. We employ real data sets to demonstrate the validity of AIR.
Sequential Network Change Detection with Its Applications to Ad Impact Relation Analysis
ICDM
2012
摘要
We are concerned with the issue of tracking changes of variable dependencies from multivariate time series. Conventionally, this issue has been addressed in the batch scenario where the whole data set is given at once, and the change detection must be done in a retrospective way. This paper addresses this issue in a sequential scenario where multivariate data are sequentially input and the detection must be done in a sequential fashion. We propose a new method for sequential tracking of variable dependencies. In it we employ a Bayesian network as a representation of variable dependencies. The key ideas of our method are, 1) we extend the theory of dynamic model selection (DMS), which has been developed in the batch-learning scenario, into the sequential setting, and apply it to our issue, 2) we conduct the change detection sequentially using dynamic programming per a window where we employ the Hoeffding's bound to automatically determine the window size. We empirically demonstrate that our proposed method is able to perform change detection more efficiently than a conventional batch method. Further, we give a new framework of an application of variable dependency change detection, which we call Ad Impact Relation analysis (AIR). In it, we detect the time point when a commercial message advertisement has given an impact on the market and effectively visulaize the impact through network changes. We employ real data sets to demonstrate the validity of AIR.
Model Change Detection With the MDL Principle
IEEE Transactions on Information Theory
2018
摘要
We are concerned with the issue of detecting model changes in probability distributions. We specifically consider the strategies based on the minimum description length (MDL) principle. We theoretically analyze their basic performance from the two aspects: data compression and hypothesis testing. From the view of data compression, we derive a new bound on the minimax regret for model changes. Here, the mini–max regret is defined as the minimum of the worst-case code-length relative to the least normalized maximum likelihood code-length over all model changes. From the view of hypothesis testing, we reduce the model change detection into a simple hypothesis testing problem. We thereby derive upper bounds on error probabilities for the MDL-based model change test. The error probabilities are valid for finite sample size and are related to the information-theoretic complexity as well as the discrepancy measure of the hypotheses to be tested.
Detecting changes of clustering structures using normalized maximum likelihood coding
KDD
2012
摘要
We are concerned with the issue of detecting changes of clustering structures from multivariate time series. From the viewpoint of the minimum description length(MDL) principle, we propose an algorithm that tracks changes of clustering structures so that the sum of the code-length for data and that for clustering changes is minimum. Here we employ a Gaussian mixture model(GMM) as representation of clustering, and compute the code-length for data sequences using the normalized maximum likelihood (NML) coding. The proposed algorithm enables us to deal with clustering dynamics including merging, splitting, emergence, disappearance of clusters from a unifying view of the MDL principle. We empirically demonstrate using artificial data sets that our proposed method is able to detect cluster changes significantly more accurately than an existing statistical-test based method and AIC/BIC-based methods. We further use real customers' transaction data sets to demonstrate the validity of our algorithm in market analysis. We show that it is able to detect changes of customer groups, which correspond to changes of real market environments.
Flexible Log File Parsing using Hidden Markov Models
ICNLP 2019
2019
摘要
We aim to model unknown file processing. As the content of log files often evolves over time, we established a dynamic statistical model which learns and adapts processing and parsing rules. First, we limit the amount of unstructured text by focusing only on those frequent patterns which lead to the desired output table similar to Vaarandi [10]. Second, we transform the found frequent patterns and the output stating the parsed table into a Hidden Markov Model (HMM). We use this HMM as a specific, however, flexible representation of a pattern for log file processing. With changes in the raw log file distorting learned patterns, we aim the model to adapt automatically in order to maintain high quality output. After training our model on one system type, applying the model and the resulting parsing rule to a different system with slightly different log file patterns, we achieve an accuracy over 99%.
With Semantics and Hidden Markov Models to an Adaptive Log File Parser
2019
摘要
We aim to model an adaptive log file parser. As the content of log files often evolves over time, we established a dynamic statistical model which learns and adapts processing and parsing rules. First, we limit the amount of unstructured text by clustering based on semantics of log file lines. Next, we only take the most relevant cluster into account and focus only on those frequent patterns which lead to the desired output table similar to Vaarandi [10]. Furthermore, we transform the found frequent patterns and the output stating the parsed table into a Hidden Markov Model (HMM). We use this HMM as a specific, however, flexible representation of a pattern for log file parsing to maintain high quality output. After training our model on one system type and applying it to a different system with slightly different log file patterns, we achieve an accuracy over 99.99%.
Finding Surprisingly Frequent Patterns of Variable Lengths in Sequence Data
SDM
2016
摘要
We address the problem of finding ‘surprising’ patterns of variable length in sequence data, where a surprising pattern is defined as a subsequence of a longer sequence, whose observed frequency is statistically significant with respect to a given distribution. Finding statistically significant patterns in sequence data is the core task in some interesting applications such as Biological motif discovery and anomaly detection. We show that the presence of few ‘true’ surprising patterns in the data could cause a large number of highlycorrelated patterns to stand statistically significant just because of those few significant patterns. Our approach to solving the ‘redundant patterns’ problem is based on capturing the dependencies between patterns through an ‘explain’ relationship where a set of patterns can explain the statistical significance of another pattern. This allows us to address the problem of redundancy by choosing a few ‘core’ patterns which explain the significance of all other significant patterns. We propose a greedy algorithm for efficiently finding an approximate core pattern set of minimum size. Using both synthetic and real-world sequential data, chosen from different domains including Medicine and Bioinformatics, we show that the proposed notion of core patterns very closely matches the notion of ‘true’ surprising patterns in
Spatio-temporal factorization of log data for understanding network events
IEEE INFOCOM 2014 - IEEE Conference on Computer Communications
2014
摘要
Understanding the impacts and patterns of network events such as link flaps or hardware errors is crucial for diagnosing network anomalies. In large production networks, analyzing the log messages that record network events has become a challenging task due to the following two reasons. First, the log messages are composed of unstructured text messages generated by vendor-specific rules. Second, network equipment such as routers, switches, and RADIUS severs generate various log messages induced by network events that span across several geographical locations, network layers, protocols, and services. In this paper, we have tackled these obstacles by building two novel techniques: statistical template extraction (STE) and log tensor factorization (LTF). STE leverages a statistical clustering technique to automatically extract primary templates from unstructured log messages. LTF aims to build a statistical model that captures spatial-temporal patterns of log messages. Such spatial-temporal patterns provide useful insights into understanding the impacts and root cause of hidden network events. This paper first formulates our problem in a mathematical way. We then validate our techniques using massive amount of network log messages collected from a large operating network. We also demonstrate several case studies that validate the usefulness of our technique.
Understanding university campus network reliability characteristics using a big data analytics tool
2015 11th International Conference on the Design of Reliable Communication Networks (DRCN)
2015
摘要
Understanding the health of a network via offline outage and failure analysis is important to assess the availability of the network and understand the pattern of failures and the root cause of them for the future network reliability improvement. In this paper, we perform a university campus network outage and failure analysis (UMKC access network) that has not been investigated well due to the lack of effective methodologies. We used Splunk, one of the big data analysis tools, to effectively handle the sheer amount of node outage and link failure log data and topology information. We investigate the network reliability characteristics via SNMP and syslog data, causes of failures, and their impact. Our study shows that the general characteristics of the different layers are very distinct from each other and the wireless network is less reliable compared to the wired network and is affected by the performance of the wired network.
Understanding the health of an access network
2011 Third International Conference on Ubiquitous and Future Networks (ICUFN)
2011
摘要
Understanding the health of a network via failure and outage analysis is important to assess the availability of a network, identify problem areas for network availability improvement, and model the exact network behavior. However, there has been little failure measurement and analysis work on access networks. In this paper, we carry out an in-depth outage and failure analysis of a university campus network using a rich set of node outage and link failure data and topology information. We investigate the attributes of hardware/software and misconfiguration problems of the networks, the relation of link failure and node outage, the node availability, and the correlations between layers of a hierarchical network. Our study shows that the general characteristics of the campus network are very distinct from backbone networks, and the hierarchical architecture and redundancy of the network design is fairly effective in the network health.
Errors and Faults
2015
摘要
Understanding the behavior of failures in large-scale systems is important in order to design techniques to tolerate them. Reliability knowledge of resources can be used in numerous ways by scientist of systems administrators: (1) it can be used to improve the quality of service of the machine; (2) to reduce performance loss due to unexpected failures either by reliability-aware scheduling or by reliability-aware checkpointing; (3) to design more resilient applications, programming models or machines in the future. This chapter focuses on offering an overview of failures observed in real large-scale systems and their characteristics, with an emphasis on modeling, detection, and prediction.