An IRL-based malware adversarial generation method to evade anti-malware engines

Abstract In order to reduce the risk of malware, researchers proposed various malware detection methods, in which the machine learning-based method has been paid more and more attention. However, malware developers used a variety of countermeasures to evade detection. For example, they may generate so-called adversarial examples to interfere with machine-learning-based detectors. An adversarial example is one that makes changes to the malware so that the generated malware can evade detection while retaining the malicious functionality. In the complex adversarial environment, only the in-depth analysis of the adversarial code can comprehensively improve the detection level of the detector. In this work, we used improved reinforcement learning to generate adversarial examples. The method accepts malicious code samples as input, and takes detection engine and feature extractor as the environment, to output several malicious samples that can avoid the detection by adjusting each detection results. Compared with the existing methods based on reinforcement learning, our method can generate reward function automatically without manual setting, which greatly improves the flexibility of the model. We compared the effectiveness of our algorithm with other methods in some of the literature on a set of portable executable files (PEs). Experimental results show that our algorithm is more flexible and effective.