Malware Detection Using Machine Learning and Deep Learning

Research shows that over the last decade, malware has been growing exponentially, causing substantial financial losses to various organizations. Different anti-malware companies have been proposing solutions to defend attacks from these malware. The velocity, volume, and the complexity of malware are posing new challenges to the anti-malware community. Current state-of-the-art research shows that recently, researchers and anti-virus organizations started applying machine learning and deep learning methods for malware analysis and detection. We have used opcode frequency as a feature vector and applied unsupervised learning in addition to supervised learning for malware classification. The focus of this tutorial is to present our work on detecting malware with 1) various machine learning algorithms and 2) deep learning models. Our results show that the Random Forest outperforms Deep Neural Network with opcode frequency as a feature. Also in feature reduction, Deep Auto-Encoders are overkill for the dataset, and elementary function like Variance Threshold perform better than others. In addition to the proposed methodologies, we will also discuss the additional issues and the unique challenges in the domain, open research problems, limitations, and future directions.

[1]  HanJiawei,et al.  Cloud-based malware detection for evolving data streams , 2008 .

[2]  Yuval Elovici,et al.  Unknown Malcode Detection Using OPCODE Representation , 2008, EuroISI.

[3]  Yanfang Ye,et al.  IMDS: intelligent malware detection system , 2007, KDD '07.

[4]  Mohit Sewak,et al.  An investigation of a deep learning based malware detection system , 2018, ARES.

[5]  Bhavani M. Thuraisingham,et al.  Cloud-based malware detection for evolving data streams , 2011, ACM Trans. Manag. Inf. Syst..

[6]  Yanfang Ye,et al.  DL 4 MD : A Deep Learning Framework for Intelligent Malware Detection , 2016 .

[7]  Sanjay Kumar Sahay,et al.  Grouping the executables to detect malware with high accuracy , 2016, ArXiv.

[8]  Tao Li,et al.  An intelligent PE-malware detection system based on association mining , 2008, Journal in Computer Virology.

[9]  Haibo He,et al.  ADASYN: Adaptive synthetic sampling approach for imbalanced learning , 2008, 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence).

[10]  Juan Caballero,et al.  Driving in the Cloud: An Analysis of Drive-by Download Operations and Abuse Reporting , 2013, DIMVA.

[11]  S. Sitharama Iyengar,et al.  A Survey on Malware Detection Using Data Mining Techniques , 2017, ACM Comput. Surv..

[12]  Igor Santos,et al.  Opcode sequences as representation of executables for data-mining-based unknown malware detection , 2013, Inf. Sci..

[13]  Mahmood Yousefi-Azar,et al.  Autoencoder-based feature learning for cyber security applications , 2017, 2017 International Joint Conference on Neural Networks (IJCNN).

[14]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[15]  Mohit Sewak,et al.  Comparison of Deep Learning and the Classical Machine Learning Algorithm for the Malware Detection , 2018, 2018 19th IEEE/ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing (SNPD).

[16]  Edward Raff,et al.  What can N-grams learn for malware detection? , 2017, 2017 12th International Conference on Malicious and Unwanted Software (MALWARE).

[17]  Sanjay Kumar Sahay,et al.  An effective approach for classification of advanced malware with high accuracy , 2016, ArXiv.

[18]  Alva Erwin,et al.  Analysis of Machine learning Techniques Used in Behavior-Based Malware Detection , 2010, 2010 Second International Conference on Advances in Computing, Control, and Telecommunication Technologies.

[19]  Yong Chen,et al.  Automatic malware categorization using cluster ensemble , 2010, KDD.

[20]  Nathan S. Netanyahu,et al.  DeepSign: Deep learning for automatic malware signature generation and classification , 2015, 2015 International Joint Conference on Neural Networks (IJCNN).