A new connection degree calculation and measurement method for large scale network monitoring

Abstract Traffic pattern characteristics monitoring is useful for abnormal behavior detection and network management. In this paper, we develop a framework for connection degree calculation and measurement in high-speed networks. The bi-directional traffic flow model is employed to aggregate traffic packets, which can reduce the number of flow records and capture user's alternation behavior characteristics. The first order connection degree and joint correlation degree are selected as the features to capture the characteristics of traffic profiles. To perform careful traffic inspection and attack detection, not only the abnormal changes of a single traffic feature but also the correlations between the features are analyzed in the new framework. First, the symmetry of in and out connection degrees is analyzed. And we found that incomplete flows are an important information source for abnormal behavior detection. Second, joint correlation degree can characterize the user's communication profiles and their behavior dynamics, which are employed to perform abnormal detection using measurements based on Renyi cross entropy. Finally, the reversible degree sketch is employed for querying abnormal traffic pattern sources for real-time traffic management. The experimental results based on actual traffic traces collected from Northwest Regional Center of CERNET (China Education and Research Network) show the efficiency of the proposed method. The method based on Renyi entropy can detect abnormal changing points correctly. FNR of the reversible sketch for locating abnormal sources is below 4% and time complexity is constant and less than 4 s, which is critical for real-time traffic monitoring.

[1]  Claude E. Shannon,et al.  A mathematical theory of communication , 1948, MOCO.

[2]  Paul Barford,et al.  A signal analysis of network traffic anomalies , 2002, IMW '02.

[3]  Nevil Brownlee,et al.  Passive measurement of one-way and two-way flow lifetimes , 2007, CCRV.

[4]  Donald F. Towsley,et al.  Worm propagation modeling and analysis under dynamic quarantine defense , 2003, WORM '03.

[5]  Abhishek Kumar,et al.  Joint data streaming and sampling techniques for detection of super sources and destinations , 2005, IMC '05.

[6]  Noga Alon,et al.  The Space Complexity of Approximating the Frequency Moments , 1999 .

[7]  Zhi-Li Zhang,et al.  Profiling internet backbone traffic: behavior models and applications , 2005, SIGCOMM '05.

[8]  Divesh Srivastava,et al.  Holistic UDAFs at streaming speeds , 2004, SIGMOD '04.

[9]  Renata Teixeira,et al.  Traffic classification on the fly , 2006, CCRV.

[10]  Ramesh Govindan,et al.  Detection and identification of network anomalies using sketch subspaces , 2006, IMC '06.

[11]  Martin May,et al.  Impact of packet sampling on anomaly detection metrics , 2006, IMC '06.

[12]  Carsten Lund,et al.  Flow sampling under hard resource constraints , 2004, SIGMETRICS '04/Performance '04.

[13]  Ming-Yang Kao,et al.  Reversible sketches: enabling monitoring and analysis over high-speed data streams , 2007, TNET.

[14]  A. L. Narasimha Reddy,et al.  A study of analyzing network traffic as images in real-time , 2005, Proceedings IEEE 24th Annual Joint Conference of the IEEE Computer and Communications Societies..

[15]  Dong Xiang,et al.  Information-theoretic measures for anomaly detection , 2001, Proceedings 2001 IEEE Symposium on Security and Privacy. S&P 2001.

[16]  Xiapu Luo,et al.  Detecting stealthy P2P botnets using statistical traffic fingerprints , 2011, 2011 IEEE/IFIP 41st International Conference on Dependable Systems & Networks (DSN).

[17]  Hui Zang,et al.  Is sampled data sufficient for anomaly detection? , 2006, IMC '06.

[18]  Albert G. Greenberg,et al.  Network anomography , 2005, IMC '05.

[19]  Donald F. Towsley,et al.  The monitoring and early detection of Internet worms , 2005, IEEE/ACM Transactions on Networking.

[20]  Donald F. Towsley,et al.  Detecting anomalies in network traffic using maximum entropy estimation , 2005, IMC '05.

[21]  Wolfgang John,et al.  Analysis of internet backbone traffic and header anomalies observed , 2007, IMC '07.

[22]  James Won-Ki Hong,et al.  A flow-based method for abnormal network traffic detection , 2004, 2004 IEEE/IFIP Network Operations and Management Symposium (IEEE Cat. No.04CH37507).

[23]  Carsten Lund,et al.  Estimating flow distributions from sampled flow statistics , 2003, SIGCOMM '03.

[24]  Claudio Narduzzi,et al.  Detection of Anomalous Behaviors in Networks from Traffic Measurements , 2006, 2006 IEEE Instrumentation and Measurement Technology Conference Proceedings.

[25]  Carsten Lund,et al.  Properties and prediction of flow statistics from sampled packet streams , 2002, IMW '02.

[26]  Yuan-Cheng Lai,et al.  Application classification using packet size distribution and port association , 2009, J. Netw. Comput. Appl..

[27]  Marina Thottan,et al.  Anomaly detection in IP networks , 2003, IEEE Trans. Signal Process..

[28]  Balachander Krishnamurthy,et al.  Sketch-based change detection: methods, evaluation, and applications , 2003, IMC '03.

[29]  Vyas Sekar,et al.  An empirical evaluation of entropy-based traffic anomaly detection , 2008, IMC '08.

[30]  Albert G. Greenberg,et al.  Combining routing and traffic data for detection of IP forwarding anomalies , 2004, SIGMETRICS '04/Performance '04.

[31]  Vern Paxson.  End-to-end internet packet dynamics , 1999, TNET.

[32]  Matthew V. Mahoney,et al.  Network traffic anomaly detection based on packet bytes , 2003, SAC '03.

[33]  M. Frans Kaashoek,et al.  Proceedings of the General Track: 2003 Usenix Annual Technical Conference Role Classification of Hosts within Enterprise Networks Based on Connection Patterns , 2022 .

[34]  Donald F. Towsley,et al.  An information-theoretic approach to network monitoring and measurement , 2005, IMC '05.

[35]  Eddie Kohler,et al.  Observed structure of addresses in IP traffic , 2006, TNET.

[36]  Patrick D. McDaniel,et al.  Analysis of Communities of Interest in Data Networks , 2005, PAM.

[37]  C. E. SHANNON,et al.  A mathematical theory of communication , 1948, MOCO.

[38]  Hari Balakrishnan,et al.  Fast portscan detection using sequential hypothesis testing , 2004, IEEE Symposium on Security and Privacy, 2004. Proceedings. 2004.

[39]  Jake D. Brutlag,et al.  Aberrant Behavior Detection in Time Series for Network Monitoring , 2000, LISA.

[40]  Tao Qin,et al.  A Data Streaming Method for Monitoring Host Connection Degrees of High-Speed Links , 2011, IEEE Transactions on Information Forensics and Security.

[41]  Hui Liu,et al.  A Peer-To-Peer Traffic Identification Method Using Machine Learning , 2007, 2007 International Conference on Networking, Architecture, and Storage (NAS 2007).

[42]  V. Rao Vemuri,et al.  Adaptive anomaly detection with evolving connectionist systems , 2007, J. Netw. Comput. Appl..

[43]  Patrick D. McDaniel,et al.  Enterprise Security: A Community of Interest Based Approach , 2006, NDSS.

[44]  Qiang Chen,et al.  Computer intrusion detection through EWMA for autocorrelated and uncorrelated data , 2003, IEEE Trans. Reliab..