Using Formal Grammar and Genetic Operators to Evolve Malware

In this paper, we leverage the concepts of formal grammar and genetic operators to evolve malware. As a case study, we take COM infectors and design their formal grammar with production rules in the BNF form. The chromosome (abstract representation) of an infector consists of genes (production rules). The code generator uses these production rules to derive the source code. The standard genetic operators --- crossover and mutation --- are applied to evolve population. The results of our experiments show that the evolved population contains a significant proportion of valid COM infectors. Moreover, approximately 7% of the evolved malware evade detection by COTS anti-virus software.