Improving NFA-Based Signature Matching Using Ordered Binary Decision Diagrams

Network intrusion detection systems (NIDS) make extensive use of regular expressions as attack signatures. Internally, NIDS represent and operate these signatures using finite automata. Existing representations of finite automata present a well-known time-space tradeoff: Deterministic automata (DFAs) provide fast matching but are memory intensive, while non-deterministic automata (NFAs) are space-efficient but are several orders of magnitude slower than DFAs. This time/space tradeoff has motivated much recent research, primarily with a focus on improving the space-efficiency of DFAs, often at the cost of reducing their performance. This paper presents NFA-OBDDs, a symbolic representation of NFAs that retains their space-efficiency while improving their time-efficiency. Experiments using Snort HTTP and FTP signature sets show that an NFA-OBDD-based representation of regular expressions can outperform traditional NFAs by up to three orders of magnitude and is competitive with a variant of DFAs, while still remaining as compact as NFAs.

[1]  John W. Lockwood,et al.  Fast and Scalable Pattern Matching for Network Intrusion Detection Systems , 2006, IEEE Journal on Selected Areas in Communications.

[2]  George Varghese,et al.  Deterministic memory-efficient string matching algorithms for intrusion detection , 2004, IEEE INFOCOM 2004.

[3]  Wayne Luk,et al.  Bitwise optimised CAM for network intrusion detection systems , 2005, International Conference on Field Programmable Logic and Applications, 2005..

[4]  Jonathan S. Turner,et al.  Advanced algorithms for fast and scalable deep packet inspection , 2006, 2006 Symposium on Architecture For Networking And Communications Systems.

[5]  Vern Paxson,et al.  Active mapping: resisting NIDS evasion without altering traffic , 2003, 2003 Symposium on Security and Privacy, 2003..

[6]  Christopher R. Clark,et al.  Scalable pattern matching for high speed networks , 2004, 12th Annual IEEE Symposium on Field-Programmable Custom Computing Machines.

[7]  Nen-Fu Huang,et al.  A fast string-matching algorithm for network processor-based intrusion detection system , 2004, TECS.

[8]  Alfred V. Aho,et al.  Efficient string matching , 1975, Commun. ACM.

[9]  John W. Lockwood,et al.  Deep packet inspection using parallel bloom filters , 2004, IEEE Micro.

[10]  Viktor K. Prasanna,et al.  Fast Regular Expression Matching Using FPGAs , 2001, The 9th Annual IEEE Symposium on Field-Programmable Custom Computing Machines (FCCM'01).

[11]  Patrick Crowley,et al.  A hybrid finite automaton for practical deep packet inspection , 2007, CoNEXT '07.

[12]  Somesh Jha,et al.  Deflating the big bang: fast and scalable deep packet inspection with extended finite automata , 2008, SIGCOMM '08.

[13]  Martin Roesch,et al.  Snort - Lightweight Intrusion Detection for Networks , 1999 .

[14]  Marco Platzner,et al.  Field Programmable Logic and Application , 2004, Lecture Notes in Computer Science.

[15]  Patrick Crowley,et al.  Algorithms to accelerate multiple regular expressions matching for deep packet inspection , 2006, SIGCOMM.

[16]  Somesh Jha,et al.  XFA: Faster Signature Matching with Extended Automata , 2008, 2008 IEEE Symposium on Security and Privacy (sp 2008).

[17]  Patrick Crowley,et al.  Efficient regular expression evaluation: theory to practice , 2008, ANCS '08.

[18]  Somesh Jha,et al.  Backtracking Algorithmic Complexity Attacks against a NIDS , 2006, 2006 22nd Annual Computer Security Applications Conference (ACSAC'06).

[19]  Helen J. Wang,et al.  Shield: vulnerability-driven network filters for preventing known vulnerability exploits , 2004, SIGCOMM 2004.

[20]  Chen-Nee Chuah,et al.  FIREMAN: a toolkit for firewall modeling and analysis , 2006, 2006 IEEE Symposium on Security and Privacy (S&P'06).

[21]  Emmanuel Fleury,et al.  An MTIDD Based Firewall , 2004, Telecommun. Syst..

[22]  D UllmanJeffrey,et al.  Introduction to automata theory, languages, and computation, 2nd edition , 2001 .

[23]  Vern Paxson,et al.  Enhancing byte-level network intrusion detection signatures with context , 2003, CCS '03.

[24]  Patrick Crowley,et al.  An improved algorithm to accelerate regular expression evaluation , 2007, ANCS '07.

[25]  Patrick Crowley,et al.  Data structures, algorithms and architectures for efficient regular expression evaluation , 2009 .

[26]  Hao Wang,et al.  Towards automatic generation of vulnerability-based signatures , 2006, 2006 IEEE Symposium on Security and Privacy (S&P'06).

[27]  Thomas Henry Ptacek,et al.  Insertion, Evasion, and Denial of Service: Eluding Network Intrusion Detection , 1998 .

[28]  Murray Hill,et al.  Yacc: Yet Another Compiler-Compiler , 1978 .

[29]  Joshua D. Guttman,et al.  Rigorous automated network security management , 2005, International Journal of Information Security.

[30]  Mohamed G. Gouda,et al.  Firewall design: consistency, completeness, and compactness , 2004, 24th International Conference on Distributed Computing Systems, 2004. Proceedings..

[31]  Shin-ichi Minato,et al.  Zero-Suppressed BDDs for Set Manipulation in Combinatorial Problems , 1993, 30th ACM/IEEE Design Automation Conference.

[32]  Jeffrey D. Ullman,et al.  Introduction to Automata Theory, Languages and Computation , 1979 .

[33]  Randal E. Bryant,et al.  Graph-Based Algorithms for Boolean Function Manipulation , 1986, IEEE Transactions on Computers.

[34]  Mark Handley,et al.  Network Intrusion Detection: Evasion, Traffic Normalization, and End-to-End Protocol Semantics , 2001, USENIX Security Symposium.

[35]  Scott Hazelhurst,et al.  A Reconfigurable Approach to Packet Filtering , 2001, FPL.

[36]  Ron K. Cytron,et al.  A Scalable Architecture For High-Throughput Regular-Expression Pattern Matching , 2006, 33rd International Symposium on Computer Architecture (ISCA'06).

[37]  Dionisios N. Pnevmatikatos,et al.  Fast, Large-Scale String Match for a 10Gbps FPGA-Based Network Intrusion Detection System , 2003, FPL.

[38]  Randy Smith,et al.  Efficient signature matching with multiple alphabet compression tables , 2008, SecureComm.

[39]  Sotiris Ioannidis,et al.  Gnort: High Performance Network Intrusion Detection Using Graphics Processors , 2008, RAID.

[40]  Somesh Jha,et al.  Language-based generation and evaluation of NIDS signatures , 2005, 2005 IEEE Symposium on Security and Privacy (S&P'05).

[41]  Brad L. Hutchings,et al.  Assisting network intrusion detection with reconfigurable hardware , 2002, Proceedings. 10th Annual IEEE Symposium on Field-Programmable Custom Computing Machines.

[42]  Timothy Sherwood,et al.  A high throughput string matching architecture for intrusion detection and prevention , 2005, 32nd International Symposium on Computer Architecture (ISCA'05).

[43]  Laxmi N. Bhuyan,et al.  Compiling PCRE to FPGA for accelerating SNORT IDS , 2007, ANCS '07.

[44]  Jeffrey D. Ullman,et al.  The compilation of regular expressions into integrated circuits , 1980, 21st Annual Symposium on Foundations of Computer Science (sfcs 1980).

[45]  George Varghese,et al.  Curing regular expressions matching algorithms from insomnia, amnesia, and acalculia , 2007, ANCS '07.

[46]  Ken Thompson,et al.  Programming Techniques: Regular expression search algorithm , 1968, Commun. ACM.

[47]  Edmund M. Clarke,et al.  Symbolic Model Checking: 10^20 States and Beyond , 1990, Inf. Comput..

[48]  Scott Hazelhurst,et al.  BINARY DECISION DIAGRAM REPRESENTATIONS OF FIREWALL AND ROUTER ACCESS LISTS , 1998 .

[49]  Alfred V. Aho,et al.  Optimal code generation for expression trees , 1975, STOC.

[50]  T. V. Lakshman,et al.  Fast and memory-efficient regular expression matching for deep packet inspection , 2006, 2006 Symposium on Architecture For Networking And Communications Systems.