Expressive, Efficient and Obfuscation Resilient Behavior Based IDS

Behavior based intrusion detection systems (BIDS) offer the only effective solution against modern malware. While dynamic BIDS have obvious advantages, their success hinges upon three interrelated factors: signature expressiveness, vulnerability to behavioral obfuscation and run-time efficiency of signature matching. To achieve higher signature expressiveness, a new approach for formal specification of the malicious functionalities based on abstract activity diagrams (AD) which incorporate multiple realizations of the specified functionality. We analyzed both inter and intra-process behavioral obfuscation techniques that can compromise existing BIDS. As a solution, we proposed specification generalization that implies augmenting (generalizing) otherwise obfuscation prone specification into more generic, obfuscation resilient specification. We suggest colored Petri nets as a basis for functionality recognition at the system call level. We implemented a prototype IDS that has been evaluated on malicious and legitimate programs. The experimental results indicated extremely low false positives and negatives. Moreover, the IDS shows very low execution overhead and negligible overhead penalty due to anti-obfuscation generalization.