A Novel Unsupervised Dead-value Detection Method for Monitoring Indicators in Data Center

For equipment monitoring, monitoring data are continuously generated, and when the monitoring data remain constant abnormally, we believe that these are dead values. Dead values are common and dead value detection is critical to subsequent data driven intelligent analysis for equipment. However, there is no specify work to study the dead value detection problem, to the best of our knowledge. In addition, due to challenges such as confusing profiles of dead values, the huge amount of monitoring data, and nonstationarity, existing anomaly detection methods are invalid. In this paper, we propose an effective dead value detection method consisting of two steps: dead value scoring and dead value detecting. In our evaluation, we analyze the monitoring data of equipment in the data center, and summarize six representative monitoring indicators with dead values as dataset. The evaluation experiments indicate the proposed dead value detection method achieves an average F1 score of 0.93, significantly outperforming the best performing baseline detection approaches by 117.5% on average.

[1]  Yang Feng,et al.  Unsupervised Anomaly Detection via Variational Auto-Encoder for Seasonal KPIs in Web Applications , 2018, WWW.

[2]  Valentino Constantinou,et al.  Detecting Spacecraft Anomalies Using LSTMs and Nonparametric Dynamic Thresholding , 2018, KDD.

[3]  Dan Pei,et al.  Opprentice: Towards Practical and Automatic Anomaly Detection Through Machine Learning , 2015, Internet Measurement Conference.

[4]  Nhien-An Le-Khac,et al.  Collective Anomaly Detection Based on Long Short-Term Memory Recurrent Neural Networks , 2016, FDSE.

[5]  Yue Zhao,et al.  LSCP: Locally Selective Combination in Parallel Outlier Ensembles , 2018, SDM.

[6]  Dan Pei,et al.  Automatic and Generic Periodicity Adaptation for KPI Anomaly Detection , 2019, IEEE Transactions on Network and Service Management.

[7]  Jayant Kalagnanam,et al.  Multi-task Multi-modal Models for Collective Anomaly Detection , 2017, 2017 IEEE International Conference on Data Mining (ICDM).

[8]  Daniel Massey,et al.  Argus: End-to-end service anomaly detection and localization from an ISP's point of view , 2012, 2012 Proceedings IEEE INFOCOM.

[9]  Hans-Peter Kriegel,et al.  LOF: identifying density-based local outliers , 2000, SIGMOD '00.

[10]  Hao Huang,et al.  Streaming Anomaly Detection Using Randomized Matrix Sketching , 2015, Proc. VLDB Endow..

[11]  Saeed Amizadeh,et al.  Generic and Scalable Framework for Automated Time-series Anomaly Detection , 2015, KDD.

[12]  B. Rosner Percentage Points for a Generalized ESD Many-Outlier Procedure , 1983 .

[13]  Wei Sun,et al.  Robust Anomaly Detection for Multivariate Time Series through Stochastic Recurrent Neural Network , 2019, KDD.

[14]  F. E. Grubbs Sample Criteria for Testing Outlying Observations , 1950 .

[15]  Zihui Ge,et al.  Crowdsourcing service-level network event monitoring , 2010, SIGCOMM '10.

[16]  Yang Chen,et al.  Visual Analysis of Collective Anomalies Using Faceted High-Order Correlation Graphs , 2020, IEEE Transactions on Visualization and Computer Graphics.

[17]  Mohiuddin Ahmed,et al.  A survey of network anomaly detection techniques , 2016, J. Netw. Comput. Appl..