Approximate detection of machine-morphed variants of malicious programs

A morphing malware is malicious software that uses a code morphing program, or morphing engine, to transform its own code into a morphed variant. The goal of this transformation is to evade recognition by malware detectors. This dissertation proposes and evaluates a new method for detecting morphed malware variants. The method uses information about the morphing engine to recognize variants created by that engine. In particular, it is shown that implementation of the requirements of good design practices of morphing malware can be capitalized upon to efficiently discriminate programs generated by a morphing engine implementing these requirements from programs that have not been generated by the engine. Exact recognition techniques implementing this method are proposed and shown to be computationally costly. Approximate efficient variations on these techniques are then proposed and successfully evaluated to recognize programs generated by a real world morphing engine, W32. Evo1. Finally, the variation of a malware's instruction distribution underlying a probabilistic morphing engine is modeled as a Markov chain. Techniques from Markov chain theory are suggested to enable the use, for detection purposes, of the distribution of the instruction-frequency vectors of the various generations of variants of morphed malware generated by a probabilistic morphing engine.