Detecting Unknown Network Attacks Using Language Models

We propose a method for network intrusion detection based on language models such as n-grams and words. Our method proceeds by extracting these models from TCP connection payloads and applying unsupervised anomaly detection. The essential part of our approach is linear-time computation of similarity measures between language models stored in trie data structures Results of our experiments conducted on two datasets of network traffic demonstrate the importance of higher-order n-grams for detection of unknown network attacks. Our method is also suitable for language models based on words, which are more amenable in practical security applications. An implementation of our system achieved detection accuracy of over 80% with no false positives on instances of recent attacks in HTTP, FTP and SMTP traffic

[1]  C. Watkins.  Dynamic Alignment Kernels , 1999 .

[2]  L. R. Dice.  Measures of the Amount of Ecologic Association Between Species , 1945 .

[3]  Leonid Portnoy,et al.  Intrusion detection with unlabeled data using clustering , 2000 .

[4]  Martin Roesch,et al.  Snort - Lightweight Intrusion Detection for Networks , 1999 .

[5]  Philip K. Chan,et al.  An Analysis of the 1999 DARPA/Lincoln Laboratory Evaluation Data for Network Anomaly Detection , 2003, RAID.

[6]  Christopher Krügel,et al.  Service specific anomaly detection for network intrusion detection , 2002, SAC '02.

[7]  Salvatore J. Stolfo,et al.  Anomalous Payload-Based Network Intrusion Detection , 2004, RAID.

[8]  W. B. Cavnar,et al.  N-gram-based text categorization , 1994 .

[9]  Klaus-Robert Müller,et al.  From outliers to prototypes: Ordering data , 2006, Neurocomputing.

[10]  P. Laskov,et al.  Intrusion Detection in Unlabeled Data with Quarter-sphere Support Vector Machines , 2004, Prax. Inf.verarb. Kommun..

[11]  Philip K. Chan,et al.  Learning rules for anomaly detection of hostile network traffic , 2003, Third IEEE International Conference on Data Mining.

[12]  Carla Marceau,et al.  Characterizing the behavior of a program using multiple-length N-grams , 2001, NSPW '00.

[13]  Michael R. Anderberg,et al.  Cluster Analysis for Applications , 1973 .

[14]  Michael Schatz,et al.  Learning Program Behavior Profiles for Intrusion Detection , 1999, Workshop on Intrusion Detection and Network Monitoring.

[15]  Salvatore J. Stolfo,et al.  A framework for constructing features and models for intrusion detection systems , 2000, TSEC.

[16]  Bernhard Schölkopf,et al.  Dynamic Alignment Kernels , 2000 .

[17]  Michael Meier,et al.  A Model for the Semantics of Attack Signatures in Misuse Detection Systems , 2004, ISC.

[18]  M Damashek,et al.  Gauging Similarity with n-Grams: Language-Independent Categorization of Text , 1995, Science.

[19]  Edward Fredkin,et al.  Trie memory , 1960, Commun. ACM.

[20]  George Nagy,et al.  Twenty Years of Document Image Analysis in PAMI , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[21]  Rene De La Briandais.  File searching using variable length keys , 1959, IRE-AIEE-ACM Computer Conference.

[22]  Salvatore J. Stolfo,et al.  Anomalous Payload-Based Worm Detection and Signature Generation , 2005, RAID.

[23]  John McHugh,et al.  The 1998 Lincoln Laboratory IDS Evaluation , 2000, Recent Advances in Intrusion Detection.

[24]  David Moore,et al.  The Spread of the Witty Worm , 2004, IEEE Secur. Priv..

[25]  Giovanni Vigna,et al.  STATL: An Attack Language for State-Based Intrusion Detection , 2002, J. Comput. Secur..

[26]  Somesh Jha,et al.  Language-based generation and evaluation of NIDS signatures , 2005, 2005 IEEE Symposium on Security and Privacy (S&P'05).

[27]  Eleazar Eskin,et al.  The Spectrum Kernel: A String Kernel for SVM Protein Classification , 2001, Pacific Symposium on Biocomputing.

[28]  Boundary detection in tokenizing network application payload for anomaly detection , 2003 .

[29]  Nello Cristianini,et al.  Classification using String Kernels , 2000 .

[30]  Christopher Krügel,et al.  Polymorphic Worm Detection Using Structural Information of Executables , 2005, RAID.

[31]  Ching Y. Suen,et al.  n-Gram Statistics for Natural Language Understanding and Text Processing , 1979, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[32]  Stephanie Forrest,et al.  A sense of self for Unix processes , 1996, Proceedings 1996 IEEE Symposium on Security and Privacy.

[33]  Jaideep Srivastava,et al.  A Comparative Study of Anomaly Detection Schemes in Network Intrusion Detection , 2003, SDM.

[34]  Philip K. Chan,et al.  PHAD: packet header anomaly detection for identifying hostile network traffic , 2001 .

[35]  Naiwen Ye,et al.  Robustness of Canberra Metric in Computer Intrusion Detection W , 2001 .

[36]  Philip K. Chan,et al.  Learning Patterns from Unix Process Execution Traces for Intrusion Detection , 1997 .

[37]  Zhenkai Liang,et al.  Automatic generation of buffer overflow attack signatures: an approach based on program behavior models , 2005, 21st Annual Computer Security Applications Conference (ACSAC'05).

[38]  Christoph C. Michael.  Finding the vocabulary of program behavior data for anomaly detection , 2003, Proceedings DARPA Information Survivability Conference and Exposition.

[39]  Salvatore J. Stolfo,et al.  Modeling system calls for intrusion detection with dynamic window sizes , 2001, Proceedings DARPA Information Survivability Conference and Exposition II. DISCEX'01.

[40]  David Thomas,et al.  The Art in Computer Programming , 2001 .

[41]  Vern Paxson,et al.  Bro: a system for detecting network intruders in real-time , 1998, Comput. Networks.

[42]  John McHugh,et al.  Testing Intrusion detection systems: a critique of the 1998 and 1999 DARPA intrusion detection system evaluations as performed by Lincoln Laboratory , 2000, TSEC.

[43]  Barak A. Pearlmutter,et al.  Detecting intrusions using system calls: alternative data models , 1999, Proceedings of the 1999 IEEE Symposium on Security and Privacy (Cat. No.99CB36344).

[44]  H. Ross.  Principles of Numerical Taxonomy , 1964 .

[45]  Stephanie Forrest,et al.  Intrusion Detection Using Sequences of System Calls , 1998, J. Comput. Secur..

[46]  Richard Lippmann,et al.  The 1999 DARPA off-line intrusion detection evaluation , 2000, Comput. Networks.

[47]  Eleazar Eskin,et al.  A GEOMETRIC FRAMEWORK FOR UNSUPERVISED ANOMALY DETECTION: DETECTING INTRUSIONS IN UNLABELED DATA , 2002 .

[48]  Matthew V. Mahoney,et al.  Network traffic anomaly detection based on packet bytes , 2003, SAC '03.

[49]  Philip K. Chan,et al.  Learning Models of Network Traffic for Detecting Novel Attacks , 2002 .

[50]  Peter Willett,et al.  Applications of n-grams in textual information systems , 1998, J. Documentation.

[51]  Richard W. Hamming,et al.  Error detecting and error correcting codes , 1950 .