Symmetry Degree Measurement and its Applications to Anomaly Detection

Anomaly detection is an important technique used to identify patterns of unusual network behavior and keep the network under control. Today, network attacks are increasing in terms of both their number and sophistication. To avoid causing significant traffic patterns and being detected by existing techniques, many new attacks tend to involve gradual adjustment of behaviors, which always generate incomplete sessions due to their running mechanisms. Accordingly, in this work, we employ the behavior symmetry degree to profile the anomalies and further identify unusual behaviors. We first proposed a symmetry degree to identify the incomplete sessions generated by unusual behaviors; we then employ a sketch to calculate the symmetry degree of internal hosts to improve the identification efficiency for online applications. To reduce the memory cost and probability of collision, we divide the IP addresses into four segments that can be used as keys of the hash functions in the sketch. Moreover, to further improve detection accuracy, a threshold selection method is proposed for dynamic traffic pattern analysis. The hash functions in the sketch are then designed using Chinese remainder theory, which can analytically trace the IP addresses associated with the anomalies. We tested the proposed techniques based on traffic data collected from the northwest center of CERNET (China Education and Research Network); the results show that the proposed methods can effectively detect anomalies in large-scale networks.

[1]  Philippe Flajolet,et al.  Probabilistic Counting Algorithms for Data Base Applications , 1985, J. Comput. Syst. Sci..

[2]  A. Salomaa,et al.  Chinese remainder theorem: applications in computing, coding, cryptography , 1996 .

[3]  kc claffy,et al.  The architecture of CoralReef: an Internet traffic monitoring software suite , 2001 .

[4]  Carsten Lund,et al.  Properties and prediction of flow statistics from sampled packet streams , 2002, IMW '02.

[5]  Paul Barford,et al.  A signal analysis of network traffic anomalies , 2002, IMW '02.

[6]  Donald F. Towsley,et al.  Worm propagation modeling and analysis under dynamic quarantine defense , 2003, WORM '03.

[7]  Balachander Krishnamurthy,et al.  Sketch-based change detection: methods, evaluation, and applications , 2003, IMC '03.

[8]  Matthew V. Mahoney,et al.  Network traffic anomaly detection based on packet bytes , 2003, SAC '03.

[9]  Mark Crovella,et al.  Diagnosing network-wide traffic anomalies , 2004, SIGCOMM '04.

[10]  Carsten Lund,et al.  Flow sampling under hard resource constraints , 2004, SIGMETRICS '04/Performance '04.

[11]  Mark Crovella,et al.  Characterization of network-wide anomalies in traffic flows , 2004, IMC '04.

[12]  Carsten Lund,et al.  Estimating flow distributions from sampled flow statistics , 2003, SIGCOMM '03.

[13]  Mark Crovella,et al.  Mining anomalies using traffic feature distributions , 2005, SIGCOMM '05.

[14]  Graham Cormode,et al.  An improved data stream summary: the count-min sketch and its applications , 2004, J. Algorithms.

[15]  A. L. Narasimha Reddy,et al.  A study of analyzing network traffic as images in real-time , 2005, Proceedings IEEE 24th Annual Joint Conference of the IEEE Computer and Communications Societies..

[16]  Patrick D. McDaniel,et al.  Analysis of Communities of Interest in Data Networks , 2005, PAM.

[17]  Hui Zang,et al.  Is sampled data sufficient for anomaly detection? , 2006, IMC '06.

[18]  Martin May,et al.  Impact of packet sampling on anomaly detection metrics , 2006, IMC '06.

[19]  Ming-Yang Kao,et al.  Reversible sketches: enabling monitoring and analysis over high-speed data streams , 2007, TNET.

[20]  Wolfgang John,et al.  Analysis of internet backbone traffic and header anomalies observed , 2007, IMC '07.

[21]  Ni Gui.  Verification Based on Keystroke Biologic Characteristics Using Support Vector Data Description , 2008 .

[22]  G. Giorgi,et al.  Detection of Anomalous Behaviors in Networks From Traffic Measurements , 2008, IEEE Transactions on Instrumentation and Measurement.

[23]  Tao Qin,et al.  A New Data Streaming Method for Locating Hosts with Large Connection Degree , 2009, GLOBECOM 2009 - 2009 IEEE Global Telecommunications Conference.

[24]  Wei Jiang,et al.  Botnet: Survey and Case Study , 2009, 2009 Fourth International Conference on Innovative Computing, Information and Control (ICICIC).

[25]  Yang Xiao,et al.  Botnet: Classification, Attacks, Detection, Tracing, and Preventive Measures , 2009, EURASIP J. Wirel. Commun. Netw..

[26]  Luo Na.  Sketch-Based Anomalies Detection with IP Address Traceability , 2009 .

[27]  Jiankun Hu,et al.  Network Traffic Analysis and SCADA Security , 2010, Handbook of Information and Communication Security.

[28]  Tao Qin,et al.  Dynamic Feature Analysis and Measurement for Large-Scale Network Traffic Monitoring , 2010, IEEE Transactions on Information Forensics and Security.

[29]  Colin Tankard,et al.  Advanced Persistent threats and how to monitor and deter them , 2011, Netw. Secur..

[30]  Rahat Masood,et al.  SWAM: Stuxnet Worm Analysis in Metasploit , 2011, 2011 Frontiers of Information Technology.

[31]  Ying Zhang,et al.  An adaptive flow counting method for anomaly detection in SDN , 2013, CoNEXT.

[32]  Dimitris Gritzalis,et al.  The Big Four - What We Did Wrong in Advanced Persistent Threat Detection? , 2013, 2013 International Conference on Availability, Reliability and Security.

[33]  Patrick P. C. Lee,et al.  LD-Sketch: A distributed sketching design for accurate and scalable anomaly detection in network data streams , 2014, IEEE INFOCOM 2014 - IEEE Conference on Computer Communications.

[34]  Naren Ramakrishnan,et al.  Detection of stealthy malware activities with traffic causality and scalable triggering relation discovery , 2014, AsiaCCS.

[35]  Tao Qin,et al.  A new connection degree calculation and measurement method for large scale network monitoring , 2014, J. Netw. Comput. Appl..

[36]  Zhongmin Cai,et al.  Mitigating Behavioral Variability for Mouse Dynamics: A Dimensionality-Reduction-Based Approach , 2014, IEEE Transactions on Human-Machine Systems.

[37]  Jugal K. Kalita,et al.  Network Anomaly Detection: Methods, Systems and Tools , 2014, IEEE Communications Surveys & Tutorials.

[38]  Tao Qin,et al.  Robust application identification methods for P2P and VoIP traffic classification in backbone networks , 2015, Knowl. Based Syst..

[39]  Saeed Amizadeh,et al.  Generic and Scalable Framework for Automated Time-series Anomaly Detection , 2015, KDD.

[40]  M. Anandapriya,et al.  Anomaly Based Host Intrusion Detection System using semantic based system call patterns , 2015, 2015 IEEE 9th International Conference on Intelligent Systems and Control (ISCO).

[41]  Christian Callegari,et al.  On the combined use of sketches and CUSUM for Anomaly Detection , 2015, 2015 International Conference on Computing and Network Communications (CoCoNet).

[42]  Yin Bao-qun,et al.  ANOMALY ANALYSIS AND IDENTIFICATION OF BACKBONE NETWORK BASED ON SKETCH AND REGULARITY DISTRIBUTION , 2015 .

[43]  Marcin Szpyrka,et al.  An Entropy-Based Network Anomaly Detection Method , 2015, Entropy.

[44]  Jing Wang,et al.  Statistical Traffic Anomaly Detection in Time-Varying Communication Networks , 2015, IEEE Transactions on Control of Network Systems.

[45]  Wolfgang Kellerer,et al.  Anomaly Detection and Identification in Large-scale Networks based on Online Time-structured Traffic Tensor Tracking , 2016 .

[46]  Naren Ramakrishnan,et al.  Causality-based Sensemaking of Network Traffic for Android Application Security , 2016, AISec@CCS.

[47]  Danny Wen-Yaw Chung,et al.  A hardware-accelerated infrastructure for flexible sketch-based network traffic monitoring , 2016, 2016 IEEE 17th International Conference on High Performance Switching and Routing (HPSR).

[48]  Morteza Mardani,et al.  Estimating Traffic and Anomaly Maps via Network Tomography , 2014, IEEE/ACM Transactions on Networking.

[49]  Zhiyuan Zheng,et al.  Safeguarding Building Automation Networks: THE-Driven Anomaly Detector Based on Traffic Analysis , 2017, 2017 26th International Conference on Computer Communication and Networks (ICCCN).

[50]  Salvatore J. Stolfo,et al.  Anomaly Detection as a Service: Challenges, Advances, and Opportunities , 2017, Anomaly Detection as a Service.

[51]  Jing Tao,et al.  Mining Long-Term Stealthy User Behaviors on High Speed Links , 2018, IEEE INFOCOM 2018 - IEEE Conference on Computer Communications.

[52]  Viktor K. Prasanna,et al.  Sketch Acceleration on FPGA and its Applications in Network Anomaly Detection , 2018, IEEE Transactions on Parallel and Distributed Systems.

[53]  Ioannis Ch. Paschalidis,et al.  Statistical Anomaly Detection via Composite Hypothesis Testing for Markov Models , 2017, IEEE Transactions on Signal Processing.

[54]  Tao Qin,et al.  Behavior Rhythm: A New Model for Behavior Visualization and Its Application in System Security Management , 2018, IEEE Access.

[55]  Tao Qin,et al.  An Effective High Threating Alarm Mining Method for Cloud Security Management , 2018, IEEE Access.

[56]  Xiapu Luo,et al.  SkyShield: A Sketch-Based Defense System Against Application Layer DDoS Attacks , 2018, IEEE Transactions on Information Forensics and Security.

[57]  Li Ling Ko,et al.  Anomaly Detection and Attribution in Networks With Temporally Correlated Traffic , 2018, IEEE/ACM Transactions on Networking.