Application-Level Unsupervised Outlier-Based Intrusion Detection and Prevention

As cyber threats are permanently jeopardizing individuals privacy and organizations’ security, there have been several efforts to empower software applications with built-in immunity. In this paper, we present our approach to immune applications through application-level, unsupervised, outlier-based intrusion detection and prevention. Our framework allows tracking application domain objects all along the processing lifecycle. It also leverages the application business context and learns from production data, without creating any training burden on the application owner. Moreover, as our framework uses runtime application instrumentation, it incurs no additional cost on the application provider. We build a fine-grained and rich-feature application behavioral model that gets down to the method level and its invocation context. We define features to be independent from the variable structure of method invocation parameters and returned values, while preserving security-relevant information. We implemented our framework in a Java environment and evaluated it on a widely-used, enterprise-grade, and open-source ERP. We tested several unsupervised outlier detection algorithms and distance functions. Our framework achieved the best results in terms of effectiveness using the Local Outlier Factor algorithm and the Clark distance, while the average instrumentation overhead per intercepted call remains acceptable.

[1]  Okba Kazar,et al.  Network security: distributed intrusion detection system using mobile agent technology , 2016, Int. J. Commun. Networks Distributed Syst..

[2]  Sung-Hyuk Cha.  Comprehensive Survey on Distance/Similarity Measures between Probability Density Functions , 2007 .

[3]  A. Madansky.  Identification of Outliers , 1988 .

[4]  Michael Schatz,et al.  A Real-Time Intrusion Detection System Based on Learning Program Behavior , 2000, Recent Advances in Intrusion Detection.

[5]  Marianne M. Swanson,et al.  Standards for Security Categorization of Federal Information and Information Systems , 2004 .

[6]  Philip S. Yu,et al.  Outlier detection for high dimensional data , 2001, SIGMOD '01.

[7]  Weibo Gong,et al.  Anomaly detection using call stack information , 2003, 2003 Symposium on Security and Privacy, 2003..

[8]  Victoria J. Hodge,et al.  A Survey of Outlier Detection Methodologies , 2004, Artificial Intelligence Review.

[9]  Hanan El Bakkali,et al.  Toward Third-Party Immune Applications , 2017, MMM-ACNS.

[10]  Sureswaran Ramadass,et al.  Distributed Agent Based Model for Intrusion Detection System Based on Artificial Immune System , 2013 .

[11]  W. R. Buckland,et al.  Outliers in Statistical Data , 1979 .

[12]  Mário M. Freire,et al.  Applications of artificial immune systems to computer security: A survey , 2017, J. Inf. Secur. Appl..

[13]  Arthur Zimek,et al.  A Framework for Clustering Uncertain Data , 2015, Proc. VLDB Endow..

[14]  Brian Demsky,et al.  ZenIDS: Introspective Intrusion Detection for PHP Applications , 2017, 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE).

[15]  Jie Yin,et al.  NativeProtector: Protecting Android Applications by Isolating and Intercepting Third-Party Native Libraries , 2016, SEC.

[16]  Jaideep Srivastava,et al.  A Comparative Study of Anomaly Detection Schemes in Network Intrusion Detection , 2003, SDM.

[17]  Yu Lin,et al.  Application intrusion detection using language library calls , 2001, Seventeenth Annual Computer Security Applications Conference.

[18]  Michael Coates.  Creating Attack- Aware Software Applications Real-Time Defenses , 2011 .

[19]  R. Sekar,et al.  A fast automaton-based method for detecting anomalous program behaviors , 2001, Proceedings 2001 IEEE Symposium on Security and Privacy. S&P 2001.

[20]  Nicolò Perino,et al.  A framework for self-healing software systems , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[21]  Lea Viljanen.  A Survey of Application Level Intrusion Detection , 2004 .

[22]  Ellen M. Voorhees,et al.  Evaluating evaluation measure stability , 2000, SIGIR '00.

[23]  Gang Tan,et al.  NativeGuard: protecting android applications from third-party native libraries , 2014, WiSec '14.

[24]  R. Pincus.  Barnett, V., and Lewis T.: Outliers in Statistical Data. 3rd edition. J. Wiley & Sons 1994, XVII. 582 pp., £49.95 , 1995 .

[25]  Hans-Peter Kriegel,et al.  Interpreting and Unifying Outlier Scores , 2011, SDM.

[26]  Aleksandar Lazarevic,et al.  Incremental Local Outlier Detection for Data Streams , 2007, 2007 IEEE Symposium on Computational Intelligence and Data Mining.

[27]  Michael Tüxen,et al.  Transport Layer Security (TLS) and Datagram Transport Layer Security (DTLS) Heartbeat Extension , 2012, RFC.

[28]  Hans-Peter Kriegel,et al.  Angle-based outlier detection in high-dimensional data , 2008, KDD.

[29]  Robert S. Sielken.  Application Intrusion Detection , 1999 .

[30]  David A. Wagner,et al.  Intrusion detection via static analysis , 2001, Proceedings 2001 IEEE Symposium on Security and Privacy. S&P 2001.

[31]  Kai Wang,et al.  Stay in Your Cage! A Sound Sandbox for Third-Party Libraries on Android , 2016, ESORICS.

[32]  Michael Backes,et al.  Seamless In-App Ad Blocking on Stock Android , 2017, 2017 IEEE Security and Privacy Workshops (SPW).

[33]  Calvin Ko,et al.  Detecting and countering system intrusions using software wrappers , 2003, Foundations of Intrusion Tolerant Systems, 2003 [Organically Assured and Survivable Information Systems].

[34]  Jim Freeman,et al.  Outliers in Statistical Data (3rd edition) , 1995 .

[35]  Hans-Peter Kriegel,et al.  LOF: identifying density-based local outliers , 2000, SIGMOD '00.

[36]  Elena Deza,et al.  Dictionary of distances , 2006 .