PbMMD: A novel policy based multi-process malware detection

Abstract Contemporary malware makes wide use of techniques to evade popular detection approaches. Behavior-based detection is the most powerful approach to malware detection. This approach is based on system call sequences to model a malicious behavior. A recently immersed malware to defeat behavior-based detection approach is Multi-process malware. This malware is the consequence of multiple processes cooperating to fulfill a malicious task each of which performing a partition of main task and none of them shows an identifiable malicious behavior. In this paper, we have presented a new method called PbMMD for detecting Multi-process malware. In this method, we attempt to inspect the whole processes running on the system and discover collaborative processes by finding processes running along a common execution policy. Beforehand we have learned different execution policy by employing reinforcement algorithm. Finally we decide against a Multi-process malicious behavior by analyzing the cumulative behavior of identified collaborative processes.