Anatomy of a Real-Time Intrusion Prevention System

Host intrusion prevention systems for both servers and end-hosts must address the dual challenges of accuracy and performance. Researchers have mostly focused on addressing the former challenge, suggesting solutions based either on exploit- based penetration detection or anomaly-based misbehavior detection, but yet stopping short of comprehensive solutions that leverage merits of both approaches. The second challenge, however, is rarely addressed; doing so comprehensively is important since these systems can introduce substantial overhead and cause system slowdown, more so when the system load is high. We present Rootsense, a holistic and real-time intrusion prevention system that combines the merits of misbehavior- based and anomaly-based detection. Four principles govern the design and implementation of Rootsense. First, Rootsense audits events within different subsystems of the host operating system and correlates them to comprehensively capture the global system state. Second, Rootsense restricts the detection domain to root compromises only; doing so reduces run-time overhead and increases detection accuracy (root behavior is more easily modeled than user behavior). Third, Rootsense adopts a dual approach to intrusion detection - a root penetration detector detects activities that exploit system vulnerabilities to penetrate the security perimeter, and a root misbehavior detector tracks misbehavior by root processes. Fourth, Rootsense is designed to be configurable for overhead management allowing the system administrator to tune the overhead characteristics of the intrusion prevention system that affect foreground task performance. A Linux implementation of Rootsense is analyzed for both accuracy and performance, using several real-world exploits and a range of end-host and server benchmarks.

[1]  Peter G. Neumann,et al.  Computer system - Security evaluation , 1899, AFIPS National Computer Conference.

[2]  Crispan Cowan,et al.  StackGuard: Automatic Adaptive Detection and Prevention of Buffer-Overflow Attacks , 1998, USENIX Security Symposium.

[3]  R. Sekar,et al.  A fast automaton-based method for detecting anomalous program behaviors , 2001, Proceedings 2001 IEEE Symposium on Security and Privacy. S&P 2001.

[4]  Stephanie Forrest,et al.  Intrusion Detection Using Sequences of System Calls , 1998, J. Comput. Secur..

[5]  Yi Zhang,et al.  Performance Adaptation in Real-Time Intrusion Detection Systems , 2002, RAID.

[6]  Debin Gao,et al.  Behavioral Distance for Intrusion Detection , 2005, RAID.

[7]  Dorothy E. Denning,et al.  An Intrusion-Detection Model , 1987, IEEE Transactions on Software Engineering.

[8]  R. Sekar,et al.  Synthesizing Fast Intrusion Prevention/Detection Systems from High-Level Specifications , 1999, USENIX Security Symposium.

[9]  Somesh Jha,et al.  Environment-Sensitive Intrusion Detection , 2005, RAID.

[10]  Herbert Bos,et al.  Modular System Programming in MINIX 3 , 2006, login Usenix Mag..

[11]  Sushil Jajodia,et al.  An Efficient and Unified Approach to Correlating, Hypothesizing, and Predicting Intrusion Alerts , 2005, ESORICS.

[12]  T N H E R D E R, H E R B E R T B O S, B E N G R A S, P,et al.  modular system programming in MINIX 3 , .

[13]  A HofmeyrSteven,et al.  Intrusion detection using sequences of system calls , 1998 .

[14]  Niels Provos,et al.  Improving Host Security with System Call Policies , 2003, USENIX Security Symposium.

[15]  Kymie M. C. Tan,et al.  "Why 6?" Defining the operational limits of stide, an anomaly-based intrusion detector , 2002, Proceedings 2002 IEEE Symposium on Security and Privacy.

[16]  Christopher Krügel,et al.  Anomalous system call detection , 2006, TSEC.

[17]  Eugene H. Spafford,et al.  The design and implementation of tripwire: a file system integrity checker , 1994, CCS '94.

[18]  Angelos D. Keromytis,et al.  Detecting Targeted Attacks Using Shadow Honeypots , 2005, USENIX Security Symposium.

[19]  Salvatore J. Stolfo,et al.  A framework for constructing features and models for intrusion detection systems , 2000, TSEC.

[20]  Stephanie Forrest,et al.  A sense of self for Unix processes , 1996, Proceedings 1996 IEEE Symposium on Security and Privacy.

[21]  Karl N. Levitt,et al.  Execution monitoring of security-critical programs in distributed systems: a specification-based approach , 1997, Proceedings. 1997 IEEE Symposium on Security and Privacy (Cat. No.97CB36097).

[22]  Martin Roesch,et al.  Snort - Lightweight Intrusion Detection for Networks , 1999 .

[23]  Stefan Axelsson,et al.  The base-rate fallacy and the difficulty of intrusion detection , 2000, TSEC.

[24]  Matt Bishop,et al.  A Taxonomy of UNIX System and Network Vulnerabilities , 1997 .

[25]  Dae-Ki Kang,et al.  Learning classifiers for misuse and anomaly detection using a bag of system calls representation , 2005, Proceedings from the Sixth Annual IEEE SMC Information Assurance Workshop.

[26]  R. Sekar,et al.  Dataflow anomaly detection , 2006, 2006 IEEE Symposium on Security and Privacy (S&P'06).

[27]  Giovanni Vigna,et al.  STATL: An Attack Language for State-Based Intrusion Detection , 2002, J. Comput. Secur..

[28]  Samuel T. King,et al.  Backtracking intrusions , 2003, SOSP '03.

[29]  Samuel T. King,et al.  Detecting past and present intrusions through vulnerability-specific predicates , 2005, SOSP '05.

[30]  Matt Bishop,et al.  Checking for Race Conditions in File Accesses , 1996, Comput. Syst..

[31]  Karl N. Levitt,et al.  Automated detection of vulnerabilities in privileged programs by execution monitoring , 1994, Tenth Annual Computer Security Applications Conference.

[32]  Daniel F. Sterne,et al.  Confining Root Programs with Domain and Type Enforcement , 1996, USENIX Security Symposium.

[33]  Carl E. Landwehr,et al.  A taxonomy of computer program security flaws , 1993, CSUR.

[34]  Vern Paxson,et al.  Bro: a system for detecting network intruders in real-time , 1998, Comput. Networks.

[35]  Stephanie Forrest,et al.  Automated response using system-call delays , 2000 .

[36]  Rebecca Gurley Bace,et al.  Intrusion Detection , 2018, Encyclopedia of Social Network Analysis and Mining. 2nd Ed..

[37]  Richard A. Kemmerer,et al.  State Transition Analysis: A Rule-Based Intrusion Detection Approach , 1995, IEEE Trans. Software Eng..

[38]  Calton Pu,et al.  TOCTTOU vulnerabilities in UNIX-style file systems: an anatomical study , 2005, FAST'05.