Flow-level traffic measurement is required for a wide range of applications including accounting, network planning and security management. A key design challenge is how to gracefully deal with traffic surges that exhaust the resources (memory, export bandwidth or CPU) of the flow monitor. A standard solution is to do sampling (look at one out of every n packets). This is implemented in Cisco’s Netflow, a popular platform. Setting the sampling rate according to the normal traffic, however, cannot avoid overrunning available memory for flow records during abnormal situations, such as when there is a DoS attack or other security breaches. Currently available countermeasures have their own problems: (1) reject new flows when the cache is full - some legitimate new flows will not be counted; (2) export not-terminated flows to make room for new ones - this will exhaust the export bandwidth; (3) adapt the sampling rate to traffic rate - this will reduce the overall accuracy of accounting, including legitimate flows. In this paper, we propose a new counter-measure to deal with abnormal traffic conditions - adaptive flow aggregation. Often the reason for abnormal traffic conditions is due to security attacks. Fortunately, such attacks usually have some common patterns. For example, packets of DoS attacks have the same destination IP address, while traffic for worm spreading has the same source IP address. Our flow monitoring algorithm identifies these traffic clusters in real-time and aggregates these large amount of short flows into a few flows.
[1]
David Moore,et al.
The CoralReef Software Suite as a Tool for System and Network Administrators
,
2001,
LISA.
[2]
George Varghese,et al.
Automatically inferring patterns of resource consumption in network traffic
,
2003,
SIGCOMM '03.
[3]
Ratul Mahajan,et al.
Controlling high bandwidth aggregates in the network
,
2002,
CCRV.
[4]
George Varghese,et al.
Building a better NetFlow
,
2004,
SIGCOMM.
[5]
George Varghese,et al.
New directions in traffic measurement and accounting
,
2002,
CCRV.
[6]
Carsten Lund,et al.
Predicting resource usage and estimation accuracy in an IP flow measurement collection infrastructure
,
2003,
IMC '03.
[7]
David Plonka,et al.
FlowScan: A Network Traffic Flow Reporting and Visualization Tool
,
2000,
LISA.
[8]
Carsten Lund,et al.
Estimating flow distributions from sampled flow statistics
,
2005,
TNET.
[9]
David Moore,et al.
A robust system for accurate real-time summaries of internet traffic
,
2005,
SIGMETRICS '05.
[10]
Zhi-Li Zhang,et al.
Adaptive random sampling for load change detection
,
2002,
SIGMETRICS '02.