Monitoring Time-Varying Network Streams Using State-Space Models

In one embodiment, a statistical model is generated based on observed data, the observed data being associated with a network device, online parameter fitting is performed on parameters of the statistical model, and for each newly observed data value, a forecast value is generated based on the statistical model, the forecast value being a prediction of a next observed data value, a forecasting error is generated based on the forecast value and the newly observed data value, and whether the data of the network stream is abnormal is determined based on a log likelihood ratio test of the forecasting errors and a threshold value.

[1]  P. Bickel,et al.  Mathematical Statistics: Basic Ideas and Selected Topics , 1977 .

[2]  Tian Bu,et al.  Design and Evaluation of a Fast and Robust Worm Detection Algorithm , 2006, Proceedings IEEE INFOCOM 2006. 25TH IEEE International Conference on Computer Communications.

[3]  Michèle Basseville,et al.  Detection of abrupt changes: theory and application , 1993 .

[4]  Balachander Krishnamurthy,et al.  Sketch-based change detection: methods, evaluation, and applications , 2003, IMC '03.

[5]  Padhraic Smyth,et al.  Learning to detect events with Markov-modulated poisson processes , 2007, TKDD.

[6]  Kavé Salamatian,et al.  Combining filtering and statistical methods for anomaly detection , 2005, IMC '05.

[7]  Mark Crovella,et al.  Mining anomalies using traffic feature distributions , 2005, SIGCOMM '05.

[8]  Chuanhai Liu,et al.  Adaptive Thresholds , 2006 .

[9]  R. E. Kalman,et al.  A New Approach to Linear Filtering and Prediction Problems , 2002 .

[10]  E. S. Page.  CONTINUOUS INSPECTION SCHEMES , 1954 .

[11]  John M. Graybeal,et al.  New optimization and management services for 3G wireless networks using CELNET Xplorer , 2005, Bell Labs Technical Journal.

[12]  C. PillersDobler,et al.  Mathematical Statistics: Basic Ideas and Selected Topics (vol. 1, 2nd ed.) , 2002 .

[13]  D.J. Leith,et al.  Adaptive Kalman Filtering for anomaly detection in software appliances , 2008, IEEE INFOCOM Workshops 2008.

[14]  João Gama,et al.  Change Detection with Kalman Filter and CUSUM , 2006, Discovery Science.

[15]  Carl de Boor,et al.  A Practical Guide to Splines , 1978, Applied Mathematical Sciences.

[16]  C HUANHAI.  Adaptive Thresholds : Monitoring Streams of Network Counts Online , 2006 .