Static analysis of programs with application to malicious core detection

The goal of this research was to develop a method of detecting malicious code (e.g., computer viruses, worms, Trojan-horses, and time/logic bombs) and security-related vulnerabilities in system programs. The Malicious Code Filter (MCF) is a programmable static analysis tool developed for this purpose. It allows system administrators and security analysts to examine a program before installation, thereby avoiding any damage a malicious program might inflict. This dissertation has addressed several issues in malicious code detection: (1) We investigated and classified malicious code. Based on this analysis, we developed the tell-tale sign approach to distinguish malicious code from benign programs. The novel idea about tell-tale signs is that they allow us to determine whether or not a program is malicious without requiring a programmer to provide a formal specification. (2) We generalized program slicing to reason about tell-tale malicious properties. Program slicing produces a bona-fide program--a subset of the original program behaving exactly the same with respect to the realization of a specified property. By combining the tell-tale sign approach with program slicing, we can examine a small subset of a large program to conclude whether or not the program is malicious. (3) We demonstrated the capabilities of the tell-tale sign approach and program slicing to detect some common UNIX vulnerabilities. (4) We determined how our approach can be defeated and developed countermeasures--the well-behavedness check. Static analysis produces inaccurate slices on a program that has pointer overflows, out-of bounds array accesses, or self-modifying code. Well-behavedness check applies flow analysis (integer-range analysis) and verification techniques (loop invariant generation, verification condition generation, and theorem proving) to identify such problematic cases. The MCF is designed in a modular way so that it can serve as a platform for further malicious code research including dynamic analysis and testing-based detection techniques.