Dynamic Feature Analysis and Measurement for Large-Scale Network Traffic Monitoring

Measuring and monitoring the changes of network traffic patterns in large-scale networks are crucial for effective network management. In this paper, we present a framework and method for detecting and measuring the dynamic changes of the pivotal traffic patterns. A bidirectional regional flow model is established to aggregate traffic packets and extract the traffic metrics and profiles. The characteristics of the regional flows are analyzed and interesting findings are obtained. A directed graph model is applied to describe the flow metrics and six flow features are extracted to capture the dynamic changes of the flow patterns. The measurements based on Renyi entropy are developed to quantitatively monitor these changes. The experimental results based on the actual network traffic data traces show that the method presented in this paper can capture the dynamic changes of pivotal traffic patterns effectively.

[1]  Hui Zang,et al.  Is sampled data sufficient for anomaly detection? , 2006, IMC '06.

[2]  Andrew W. Moore,et al.  Traffic Classification Using a Statistical Approach , 2005, PAM.

[3]  Hui Liu,et al.  A Peer-To-Peer Traffic Identification Method Using Machine Learning , 2007, 2007 International Conference on Networking, Architecture, and Storage (NAS 2007).

[4]  Carsten Lund,et al.  Estimating flow distributions from sampled flow statistics , 2005, TNET.

[5]  Matthew V. Mahoney,et al.  Network traffic anomaly detection based on packet bytes , 2003, SAC '03.

[6]  Carsten Lund,et al.  Flow sampling under hard resource constraints , 2004, SIGMETRICS '04/Performance '04.

[7]  Nick Feamster,et al.  Geographic locality of IP prefixes , 2005, IMC '05.

[8]  Tao Qin,et al.  Dynamic Features Measurement and Analysis for Large-Scale Networks , 2008, ICC Workshops - 2008 IEEE International Conference on Communications Workshops.

[9]  M. Frans Kaashoek,et al.  Proceedings of the General Track: 2003 Usenix Annual Technical Conference Role Classification of Hosts within Enterprise Networks Based on Connection Patterns , 2022 .

[10]  Donald F. Towsley,et al.  An information-theoretic approach to network monitoring and measurement , 2005, IMC '05.

[11]  Konstantina Papagiannaki,et al.  Structural analysis of network traffic flows , 2004, SIGMETRICS '04/Performance '04.

[12]  Dong Xiang,et al.  Information-theoretic measures for anomaly detection , 2001, Proceedings 2001 IEEE Symposium on Security and Privacy. S&P 2001.

[13]  Albert G. Greenberg,et al.  Network anomography , 2005, IMC '05.

[14]  Marina Thottan,et al.  Anomaly detection in IP networks , 2003, IEEE Trans. Signal Process..

[15]  Balachander Krishnamurthy,et al.  Sketch-based change detection: methods, evaluation, and applications , 2003, IMC '03.

[16]  Martin May,et al.  Impact of packet sampling on anomaly detection metrics , 2006, IMC '06.

[17]  Nevil Brownlee,et al.  A Methodology for Finding Significant Network Hosts , 2007, 32nd IEEE Conference on Local Computer Networks (LCN 2007).

[18]  Donald F. Towsley,et al.  Detecting anomalies in network traffic using maximum entropy estimation , 2005, IMC '05.

[19]  Kathleen M. Carley,et al.  Dynamic Social Network Modeling and Analysis: Workshop Summary and Papers , 2004 .

[20]  Mark Crovella,et al.  Characterization of network-wide anomalies in traffic flows , 2004, IMC '04.

[21]  Eddie Kohler,et al.  Observed Structure of Addresses in IP Traffic , 2002, IEEE/ACM Transactions on Networking.

[22]  Carsten Lund,et al.  Properties and prediction of flow statistics from sampled packet streams , 2002, IMW '02.

[23]  Vyas Sekar,et al.  An empirical evaluation of entropy-based traffic anomaly detection , 2008, IMC '08.

[24]  Patrick D. McDaniel,et al.  Enterprise Security: A Community of Interest Based Approach , 2006, NDSS.

[25]  Guofei Gu,et al.  Worm Detection Using Local Networks , 2004 .

[26]  Claudio Narduzzi,et al.  Detection of Anomalous Behaviors in Networks from Traffic Measurements , 2006, 2006 IEEE Instrumentation and Measurement Technology Conference Proceedings.

[27]  James Won-Ki Hong,et al.  A flow-based method for abnormal network traffic detection , 2004, 2004 IEEE/IFIP Network Operations and Management Symposium (IEEE Cat. No.04CH37507).

[28]  kc claffy,et al.  The architecture of CoralReef: an Internet traffic monitoring software suite , 2001 .

[29]  Zhi-Li Zhang,et al.  Profiling internet backbone traffic: behavior models and applications , 2005, SIGCOMM '05.

[30]  Yuval Shavitt,et al.  A Simulation Study of Multi-Color Marking of TCP Aggregates , 2007 .

[31]  Renata Teixeira,et al.  Traffic classification on the fly , 2006, CCRV.

[32]  Qiang Chen,et al.  Computer intrusion detection through EWMA for autocorrelated and uncorrelated data , 2003, IEEE Trans. Reliab..

[33]  Albert G. Greenberg,et al.  Combining routing and traffic data for detection of IP forwarding anomalies , 2004, SIGMETRICS '04/Performance '04.

[34]  Vern Paxson,et al.  End-to-end Internet packet dynamics , 1997, SIGCOMM '97.

[35]  Paul Barford,et al.  A signal analysis of network traffic anomalies , 2002, IMW '02.

[36]  Nevil Brownlee,et al.  Passive measurement of one-way and two-way flow lifetimes , 2007, CCRV.

[37]  Wolfgang John,et al.  Analysis of internet backbone traffic and header anomalies observed , 2007, IMC '07.

[38]  Graham Cormode,et al.  An improved data stream summary: the count-min sketch and its applications , 2004, J. Algorithms.

[39]  Kathleen M. Carley,et al.  Toward an interoperable dynamic network analysis toolkit , 2007, Decis. Support Syst..

[40]  A. L. Narasimha Reddy,et al.  A study of analyzing network traffic as images in real-time , 2005, Proceedings IEEE 24th Annual Joint Conference of the IEEE Computer and Communications Societies..

[41]  Philippe Flajolet,et al.  Probabilistic Counting Algorithms for Data Base Applications , 1985, J. Comput. Syst. Sci..

[42]  Donald F. Towsley,et al.  The monitoring and early detection of Internet worms , 2005, IEEE/ACM Transactions on Networking.

[43]  B. Tellenbach,et al.  Impact of Traffic Mix and Packet Sampling on Anomaly Visibility , 2008, 2008 The Third International Conference on Internet Monitoring and Protection.

[44]  Kathleen M. Carley.  Dynamic Network Analysis , 2003 .

[45]  Hari Balakrishnan,et al.  Fast portscan detection using sequential hypothesis testing , 2004, IEEE Symposium on Security and Privacy, 2004. Proceedings. 2004.

[46]  Patrick D. McDaniel,et al.  Analysis of Communities of Interest in Data Networks , 2005, PAM.