A Survey of Stealth Malware Attacks, Mitigation Measures, and Steps Toward Autonomous Open World Solutions

As our professional, social, and financial existences become increasingly digitized and as our government, healthcare, and military infrastructures rely more on computer technologies, they present larger and more lucrative targets for malware. Stealth malware in particular poses an increased threat because it is specifically designed to evade detection mechanisms, spreading dormant, in the wild for extended periods of time, gathering sensitive information or positioning itself for a high-impact zero-day attack. Policing the growing attack surface requires the development of efficient anti-malware solutions with improved generalization to detect novel types of malware and resolve these occurrences with as little burden on human experts as possible. In this paper, we survey malicious stealth technologies as well as existing solutions for detecting and categorizing these countermeasures autonomously. While machine learning offers promising potential for increasingly autonomous solutions with improved generalization to new malware types, both at the network level and at the host level, our findings suggest that several flawed assumptions inherent to most recognition algorithms prevent a direct mapping between the stealth malware recognition problem and a machine learning solution. The most notable of these flawed assumptions is the closed world assumption: that no sample belonging to a class outside of a static training set will appear at query time. We present a formalized adaptive open world framework for stealth malware recognition and relate it mathematically to research from other machine learning domains.

[1]  Song Guo,et al.  Fool Me If You Can: Mimicking Attacks and Anti-Attacks in Cyberspace , 2015, IEEE Transactions on Computers.

[2]  Macia-FernandezG.,et al.  Anomaly-based network intrusion detection , 2009 .

[3]  Eric Filiol,et al.  Metamorphism, Formal Grammars and Undecidable Code Mutation , 2007 .

[4]  Jason Yosinski,et al.  Deep neural networks are easily fooled: High confidence predictions for unrecognizable images , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Shou-De Lin,et al.  Feature Engineering and Classifier Ensemble for KDD Cup 2010 , 2010, KDD 2010.

[6]  Terrance E. Boult,et al.  Towards Open World Recognition , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Luca Faust,et al.  Modern Operating Systems , 2016 .

[8]  Stephanie Forrest,et al.  A sense of self for Unix processes , 1996, Proceedings 1996 IEEE Symposium on Security and Privacy.

[9]  Xuxian Jiang,et al.  Multi-aspect profiling of kernel rootkit behavior , 2009, EuroSys '09.

[10]  Weibo Gong,et al.  Anomaly detection using call stack information , 2003, 2003 Symposium on Security and Privacy, 2003..

[11]  EMMANOUIL VASILOMANOLAKIS,et al.  Taxonomy and Survey of Collaborative Intrusion Detection , 2015, ACM Comput. Surv..

[12]  Terrance E. Boult,et al.  The Extreme Value Machine , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  Bernd Eggers Rootkits Subverting The Windows Kernel , 2016 .

[14]  Guofei Gu,et al.  Using an Ensemble of One-Class SVM Classifiers to Harden Payload-based Anomaly Detection Systems , 2006, Sixth International Conference on Data Mining (ICDM'06).

[15]  R. Chandrasekar,et al.  A Self-organized Agent-based architecture for Power-aware Intrusion Detection in wireless ad-hoc networks , 2006, 2006 International Conference on Computing & Informatics.

[16]  Shilpa Lakhina,et al.  Feature Reduction using Principal Component Analysis for Effective Anomaly – Based Intrusion Detection on NSL-KDD , 2010 .

[17]  Abhinav Srivastava,et al.  Robust signatures for kernel data structures , 2009, CCS.

[18]  Sami Khuri,et al.  ANALYSIS AND DETECTION OF METAMORPHIC COMPUTER VIRUSES , 2006 .

[19]  Taghi M. Khoshgoftaar,et al.  Intrusion detection and Big Heterogeneous Data: a Survey , 2015, Journal of Big Data.

[20]  Charu C. Aggarwal,et al.  Data Streams - Models and Algorithms , 2014, Advances in Database Systems.

[21]  Vasant Honavar,et al.  Intelligent agents for intrusion detection , 1998, 1998 IEEE Information Technology Conference, Information Environment for the Future (Cat. No.98EX228).

[22]  Morteza Amini,et al.  RT-UNNID: A practical solution to real-time network-based intrusion detection using unsupervised neural networks , 2006, Comput. Secur..

[23]  Salvatore J. Stolfo,et al.  One Class Support Vector Machines for Detecting Anomalous Windows Registry Accesses , 2003 .

[24]  Peter Tiño,et al.  Concept drift detection for online class imbalance learning , 2013, The 2013 International Joint Conference on Neural Networks (IJCNN).

[25]  Graham J. Williams,et al.  On-Line Unsupervised Outlier Detection Using Finite Mixtures with Discounting Learning Algorithms , 2000, KDD '00.

[26]  Jörg Hähner,et al.  A building block for awareness in technical systems: Online novelty detection and reaction with an application in intrusion detection , 2015, 2015 IEEE 7th International Conference on Awareness Science and Technology (iCAST).

[27]  Andrew H. Sung,et al.  Intrusion detection using neural networks and support vector machines , 2002, Proceedings of the 2002 International Joint Conference on Neural Networks. IJCNN'02 (Cat. No.02CH37290).

[28]  D. Woolley The White Paper. , 1972, British medical journal.

[29]  Mark Stamp,et al.  Metamorphic worm that carries its own morphing engine , 2013, Journal of Computer Virology and Hacking Techniques.

[30]  Anderson Rocha,et al.  Robust Fusion: Extreme Value Theory for Recognition Score Normalization , 2010, ECCV.

[31]  Jiankun Hu,et al.  A Semantic Approach to Host-Based Intrusion Detection Systems Using Contiguousand Discontiguous System Call Patterns , 2014, IEEE Transactions on Computers.

[32]  Abhinav Srivastava,et al.  Operating System Interface Obfuscation and the Revealing of Hidden Operations , 2011, DIMVA.

[33]  Joohan Lee,et al.  A survey of data mining techniques for malware detection using file features , 2008, ACM-SE 46.

[34]  Tingquan Deng,et al.  An Adaptive Weighted One-Class SVM for Robust Outlier Detection , 2016 .

[35]  Terrance E. Boult,et al.  Multi-class Open Set Recognition Using Probability of Inclusion , 2014, ECCV.

[36]  Gideon Creech,et al.  Developing a high-accuracy cross platform Host-Based Intrusion Detection System capable of reliably detecting zero-day attacks , 2014 .

[37]  Gabriel Maciá-Fernández,et al.  Anomaly-based network intrusion detection: Techniques, systems and challenges , 2009, Comput. Secur..

[38]  Vlado Keselj,et al.  N-gram-based detection of new malicious code , 2004, Proceedings of the 28th Annual International Computer Software and Applications Conference, 2004. COMPSAC 2004..

[39]  Peter Szor,et al.  HUNTING FOR METAMORPHIC , 2001 .

[40]  Nizar Bouguila,et al.  Anomaly Intrusion Detection Using Incremental Learning of an Infinite Mixture Model with Feature Selection , 2013, RSKT.

[41]  Muttukrishnan Rajarajan,et al.  Android Security: A Survey of Issues, Malware Penetration, and Defenses , 2015, IEEE Communications Surveys & Tutorials.

[42]  Emmanuel Müller,et al.  Proceedings of the ACM SIGKDD Workshop on Outlier Detection and Description , 2013, KDD 2013.

[43]  Bernhard Schölkopf,et al.  Estimating the Support of a High-Dimensional Distribution , 2001, Neural Computation.

[44]  Jun Gao,et al.  Online Adaboost-Based Parameterized Methods for Dynamic Distributed Network Intrusion Detection , 2014, IEEE Transactions on Cybernetics.

[45]  Vijay Laxmi,et al.  DroidAnalyst: Synergic App Framework for Static and Dynamic App Analysis , 2016, Recent Advances in Computational Intelligence in Defense and Security.

[46]  Yi-Min Wang,et al.  Detecting stealth software with Strider GhostBuster , 2005, 2005 International Conference on Dependable Systems and Networks (DSN'05).

[47]  Ilsun You,et al.  A Brief Survey on Rootkit Techniques in Malicious Codes , 2012, J. Internet Serv. Inf. Secur..

[48]  Salvatore J. Stolfo,et al.  A data mining framework for building intrusion detection models , 1999, Proceedings of the 1999 IEEE Symposium on Security and Privacy (Cat. No.99CB36344).

[49]  S. Nadarajah,et al.  Extreme Value Distributions: Theory and Applications , 2000 .

[50]  Priti Desai Towards an Undetectable Computer Virus , 2008 .

[51]  Charu C. Aggarwal,et al.  Data Streams: Models and Algorithms (Advances in Database Systems) , 2006 .

[52]  Jaideep Srivastava,et al.  A Comparative Study of Anomaly Detection Schemes in Network Intrusion Detection , 2003, SDM.

[53]  James Cannady Applying CMAC-based online learning to intrusion detection , 2000, Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium.

[54]  Muhammad Hussain,et al.  Feature Subset Selection for Network Intrusion Detection Mechanism Using Genetic Eigen Vectors , .

[55]  Andreas Schuster,et al.  Searching for processes and threads in Microsoft Windows memory dumps , 2006, Digit. Investig..

[56]  Jiankun Hu,et al.  Generation of a new IDS test dataset: Time to retire the KDD collection , 2013, 2013 IEEE Wireless Communications and Networking Conference (WCNC).

[57]  Jayanta K. Ghosh,et al.  Bayesian Networks and Decision Graphs, 2nd Edition by Finn V. Jensen, Thomas D. Nielsen , 2008 .

[58]  Wei Xu,et al.  Improving one-class SVM for anomaly detection , 2003, Proceedings of the 2003 International Conference on Machine Learning and Cybernetics (IEEE Cat. No.03EX693).

[59]  Anton Chuvakin,et al.  Ups and Downs of UNIX/Linux Host-Based Security Solutions , 2003, Login: The Usenix Magazine.

[60]  Tal Garfinkel,et al.  A Virtual Machine Introspection Based Architecture for Intrusion Detection , 2003, NDSS.

[61]  Mark Stamp,et al.  Opcode graph similarity and metamorphic detection , 2012, Journal in Computer Virology.

[62]  Tal Garfinkel,et al.  Shredding Your Garbage: Reducing Data Lifetime Through Secure Deallocation , 2005, USENIX Security Symposium.

[63]  Гарнаева Мария Александровна,et al.  Kaspersky security Bulletin 2013 , 2014 .

[64]  A.H. Sung,et al.  Identifying important features for intrusion detection using support vector machines and neural networks , 2003, 2003 Symposium on Applications and the Internet, 2003. Proceedings..

[65]  Mark Stamp,et al.  Hunting for undetectable metamorphic viruses , 2011, Journal in Computer Virology.

[66]  Levente Buttyán,et al.  The Cousins of Stuxnet: Duqu, Flame, and Gauss , 2012, Future Internet.

[67]  Slim Abdennadher,et al.  Enhancing one-class support vector machines for unsupervised anomaly detection , 2013, ODD '13.

[68]  Nicholas Nethercote,et al.  Valgrind: a framework for heavyweight dynamic binary instrumentation , 2007, PLDI '07.

[69]  R. Sekar,et al.  A fast automaton-based method for detecting anomalous program behaviors , 2001, Proceedings 2001 IEEE Symposium on Security and Privacy. S&P 2001.

[70]  Bhavani M. Thuraisingham,et al.  Classification and Novel Class Detection in Concept-Drifting Data Streams under Time Constraints , 2011, IEEE Transactions on Knowledge and Data Engineering.

[71]  Finn V. Jensen,et al.  Bayesian Networks and Decision Graphs , 2001, Statistics for Engineering and Information Science.

[72]  Carla E. Brodley,et al.  Approaches to Online Learning and Concept Drift for User Identification in Computer Security , 1998, KDD.

[73]  A CataniaCarlos,et al.  Automatic network intrusion detection , 2012 .

[74]  Sameer Singh,et al.  Novelty detection: a review - part 1: statistical approaches , 2003, Signal Process..

[75]  Yuxin Ding,et al.  Host-based intrusion detection using dynamic and static behavioral models , 2003, Pattern Recognit..

[76]  Salvatore J. Stolfo,et al.  Data Mining Approaches for Intrusion Detection , 1998, USENIX Security Symposium.

[77]  Terrance E. Boult,et al.  Probability Models for Open Set Recognition , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[78]  Grant Dick,et al.  Feature Selection of Intrusion Detection Data using a Hybrid Genetic Algorithm/KNN Approach , 2003, HIS.

[79]  Jonathon Shlens,et al.  Explaining and Harnessing Adversarial Examples , 2014, ICLR.

[80]  Aziz Mohaisen,et al.  Detecting and classifying method based on similarity matching of Android malware behavior with profile , 2016, SpringerPlus.

[81]  Chris North,et al.  Visual correlation of host processes and network traffic , 2005, IEEE Workshop on Visualization for Computer Security, 2005. (VizSEC 05)..

[82]  Sergio M. Savaresi,et al.  Unsupervised learning techniques for an intrusion detection system , 2004, SAC '04.

[83]  Slobodan Petrovic,et al.  Improving Effectiveness of Intrusion Detection by Correlation Feature Selection , 2010, 2010 International Conference on Availability, Reliability and Security.

[84]  Salvatore J. Stolfo,et al.  Anomalous Payload-Based Network Intrusion Detection , 2004, RAID.

[85]  Balaraman Ravindran,et al.  Adaptive network intrusion detection system using a hybrid approach , 2012, 2012 Fourth International Conference on Communication Systems and Networks (COMSNETS 2012).

[86]  Adrian Perrig,et al.  SecVisor: a tiny hypervisor to provide lifetime kernel code integrity for commodity OSes , 2007, SOSP.

[87]  Heidar A. Malki,et al.  Network Intrusion Detection System Using Neural Networks , 2008, 2008 Fourth International Conference on Natural Computation.

[88]  Somesh Jha,et al.  Automatic placement of authorization hooks in the linux security modules framework , 2005, CCS '05.

[89]  Anthony Gar-On Yeh,et al.  Outlier Detection In Large-scale Traffic Data By Naïve Bayes Method and Gaussian Mixture Model Method , 2015, IRIACV.

[90]  Taghi M. Khoshgoftaar,et al.  CLUSTERING-BASED NETWORK INTRUSION DETECTION , 2007 .

[91]  Yuan Yuan Survey on Android Rootkit , 2011 .

[92]  William A. Arbaugh,et al.  An Architecture for Specification-Based Detection of Semantic Integrity Violations in Kernel Dynamic Data , 2006, USENIX Security Symposium.

[93]  Eddie Kohler,et al.  Information flow control for standard OS abstractions , 2007, SOSP.

[94]  Andrew H. Sung,et al.  Feature Selection for Intrusion Detection with Neural Networks and Support Vector Machines , 2003 .

[95]  Philippe Beaucamps Advanced Polymorphic Techniques , 2007 .

[96]  Pavel V. Zbitskiy Code mutation techniques by means of formal grammars and automatons , 2009, Journal in Computer Virology.

[97]  Sujandharan Venkatachalam,et al.  DETECTING UNDETECTABLE COMPUTER VIRUSES , 2010 .

[98]  Somesh Jha,et al.  Detecting Manipulated Remote Call Streams , 2002, USENIX Security Symposium.

[99]  Muttukrishnan Rajarajan,et al.  Evaluation of Android Anti-malware Techniques against Dalvik Bytecode Obfuscation , 2014, 2014 IEEE 13th International Conference on Trust, Security and Privacy in Computing and Communications.

[100]  Anderson Rocha,et al.  Toward Open Set Recognition , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[101]  Dit-Yan Yeung,et al.  Parzen-window network intrusion detectors , 2002, Object recognition supported by user interaction for service robots.

[102]  Mark Stamp,et al.  Profile hidden Markov models and metamorphic virus detection , 2009, Journal in Computer Virology.

[103]  Joan Bruna,et al.  Intriguing properties of neural networks , 2013, ICLR.

[104]  Zhang Yi,et al.  A hierarchical intrusion detection model based on the PCA neural networks , 2007, Neurocomputing.

[105]  Galen C. Hunt,et al.  Detours: binary interception of Win32 functions , 1999 .

[106]  André Zúquete,et al.  Traffic classification and verification using unsupervised learning of Gaussian Mixture Models , 2015, 2015 IEEE International Workshop on Measurements & Networking (M&N).

[107]  Kien A. Hua,et al.  Decision tree classifier for network intrusion detection with GA-based feature selection , 2005, ACM Southeast Regional Conference.

[108]  Giovanni Vigna,et al.  Exploiting Execution Context for the Detection of Anomalous System Calls , 2007, RAID.

[109]  Andrew S. Miner,et al.  Anomaly intrusion detection using one class SVM , 2004, Proceedings from the Fifth Annual IEEE SMC Information Assurance Workshop, 2004..

[110]  Carlos García Garino,et al.  Automatic network intrusion detection: Current techniques and open issues , 2012, Comput. Electr. Eng..

[111]  Arun K. Pujari,et al.  New Malicious Code Detection Using Variable Length n-grams , 2006, ICISS.

[112]  Xuxian Jiang,et al.  Guest-Transparent Prevention of Kernel Rootkits with VMM-Based Memory Shadowing , 2008, RAID.

[113]  Peter Szor,et al.  The Art of Computer Virus Research and Defense , 2005 .

[114]  Joanna Rutkowska Detecting Windows Server Compromises with Patchfinder 2 , 2004 .

[115]  Ashwini Venkatesan CODE OBFUSCATION AND VIRUS DETECTION , 2009 .

[116]  Mark Stamp,et al.  Hunting for metamorphic engines , 2006, Journal in Computer Virology.

[117]  William A. Arbaugh,et al.  Copilot - a Coprocessor-based Kernel Runtime Integrity Monitor , 2004, USENIX Security Symposium.

[118]  Christopher M. Bishop,et al.  Pattern Recognition and Machine Learning (Information Science and Statistics) , 2006 .

[119]  Eugene H. Spafford,et al.  The design and implementation of tripwire: a file system integrity checker , 1994, CCS '94.

[120]  Arati Baliga,et al.  Detecting Kernel-Level Rootkits Using Data Structure Invariants , 2011, IEEE Transactions on Dependable and Secure Computing.

[121]  Stefan Axelsson,et al.  Intrusion Detection Systems: A Survey and Taxonomy , 2002 .

[122]  Stephanie Forrest,et al.  Intrusion Detection Using Sequences of System Calls , 1998, J. Comput. Secur..

[123]  Wanlei Zhou,et al.  Modeling malicious activities in cyber space , 2015, IEEE Network.

[124]  Leonid Portnoy,et al.  Intrusion detection with unlabeled data using clustering , 2000 .

[125]  Wei-Yang Lin,et al.  Intrusion detection by machine learning: A review , 2009, Expert Syst. Appl..

[126]  Vern Paxson,et al.  Outside the Closed World: On Using Machine Learning for Network Intrusion Detection , 2010, 2010 IEEE Symposium on Security and Privacy.

[127]  Jian Ma,et al.  A new approach to intrusion detection using Artificial Neural Networks and fuzzy clustering , 2010, Expert Syst. Appl..

[128]  Terrance E. Boult,et al.  Multi-attribute spaces: Calibration for attribute fusion and similarity search , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[129]  Michal Pechoucek,et al.  Adaptive Multiagent System for Network Traffic Monitoring , 2009, IEEE Intelligent Systems.