On the Beneficial Impact of Strong Correlations for Anomaly Detection

It is now widely accepted that packet network traffic exhibits long-range dependence (LRD), and this has been shown to be harmful to network performance. LRD also reduces the effectiveness of estimators of traffic parameters. For instance, it is much harder to estimate the mean of a LRD process than that of a process with only short-term correlations. One might intuitively expect that LRD would be detrimental to most networking tasks. One important network task is anomaly detection. Anomalies often correspond to problems, for instance, denial-of-service attacks or outages, and so rapid detection is important for maintaining a reliable network. In this article we demonstrate that, counter to the above intuition, LRD is actually beneficial to the detection of anomalies, as in fact are other forms of strong correlations in the observed process. We provide both theoretical proofs and simulation examples to show that LRD in traffic measurements actually improves the probability of detection of anomalies in that traffic.

[1]  Walter Willinger,et al.  On the self-similar nature of Ethernet traffic , 1993, SIGCOMM '93.

[2]  Ilkka Norros,et al.  A storage model with self-similar input , 1994, Queueing Syst. Theory Appl..

[3]  Matthew Roughan,et al.  What does the mean mean , 2003 .

[4]  Darryl Veitch,et al.  Understanding end-to-end Internet traffic dynamics , 1998, IEEE GLOBECOM 1998 (Cat. NO. 98CH36250).

[5]  Patrice Abry,et al.  A Wavelet-Based Joint Estimator of the Parameters of Long-Range Dependence , 1999, IEEE Trans. Inf. Theory.

[6]  Matthew Roughan,et al.  Large-scale measurement and modeling of backbone Internet traffic , 2002, SPIE ITCom.

[7]  Walter Willinger,et al.  On the Self-Similar Nature of Ethernet Traffic ( extended version ) , 1995 .

[8]  Balachander Krishnamurthy,et al.  Sketch-based change detection: methods, evaluation, and applications , 2003, IMC '03.

[9]  Matthew Roughan,et al.  Measuring long-range dependence under changing traffic conditions , 1999, IEEE INFOCOM '99. Conference on Computer Communications. Proceedings. Eighteenth Annual Joint Conference of the IEEE Computer and Communications Societies. The Future is Now (Cat. No.99CH36320).

[10]  Paul Barford,et al.  A signal analysis of network traffic anomalies , 2002, IMW '02.

[11]  Matthew Roughan,et al.  IP forwarding anomalies and improving their detection using multiple data sources , 2004, NetT '04.

[12]  Jan Beran,et al.  Statistics for long-memory processes , 1994 .

[13]  V. Paxson,et al.  Wide-area traffic: the failure of Poisson modeling , 1994, SIGCOMM.

[14]  Matthew Roughan,et al.  Experience in measuring internet backbone traffic variability: Models metrics, measurements and meaning , 2003 .

[15]  Vern Paxson,et al.  An architecture for large-scale Internet measurement , 1998, IEEE Commun. Mag..

[16]  Matthias Grossglauser,et al.  On the relevance of long-range dependence in network traffic , 1996, SIGCOMM '96.

[17]  Matthew Roughan,et al.  The need for simulation in evaluating anomaly detectors , 2008, CCRV.

[18]  Albert G. Greenberg,et al.  Experience in measuring backbone traffic variability: models, metrics, measurements and meaning , 2002, IMW '02.

[19]  D. Applebaum.  Stable non-Gaussian random processes , 1995, The Mathematical Gazette.

[20]  Albert G. Greenberg,et al.  Combining routing and traffic data for detection of IP forwarding anomalies , 2004, SIGMETRICS '04/Performance '04.

[21]  Walter Willinger,et al.  Experimental queueing analysis with long-range dependent packet traffic , 1996, TNET.

[22]  Matthew J. Zekauskas,et al.  A One-way Delay Metric for IPPM , 1999, RFC.