Advanced methods for detection of malicious software

This dissertation introduces a biologically inspired model of malicious software self-replication called the Gene of Self-Replication (GSR). This model is utilized to facilitate generic detection of malware in script viruses and compiled binary executables. It incorporates a variety of important information regarding the interaction between the virus and the susceptible system, enabling a higher level of intelligent code and behavior analysis. The model provides a proactive solution for detection and elimination of previously unknown malicious code. An extensive description of the taxonomy of self-replication in malicious software is presented. It enables a systematic approach for self-replication analysis and classification based on malicious code representation and behavior investigation. Several examples of code replication analysis are given. Two methods for detection of the GSR are considered. Script viruses are treated as static objects allowing for safe analysis and detection of the GSR. A script code analysis algorithm allows us to search for propagation patterns within the script code. It is shown that GSR increases successful detection rate for previously unknown script viruses, and at the same time eliminates the need for an extensive library of available virus signatures. The second method involves a different approach for GSR detection in binary executable viruses. Due to complications with encrypted and polymorphic viral bodies, the concept of an advanced secure system monitoring agent for kernel level process behavior analysis is chosen as the source of information for GSR detection. We demonstrate that the analysis of system calls and associated arguments at the system level provides unambiguous information regarding process behavior in the replication domain. The GSR model is utilized to create a software tool for analysis and detection of self-replication within script viruses. A modified version of the model is applied to the design of a complete visual system for monitoring process behavior of the entire computational environment with an ability to intercept, detect, and disarm potentially malicious processes before they successfully complete the self-replication cycle. Finally, verification and validation of the proposed approach is presented based on a detailed design of a virtual laboratory for computer and network security research and analysis.