Application security through program obfuscation

Business models behind products such as iTunes and the Skype VoIP clients depend entirely on the secrecy of technical details of their product. Once the technical details are uncovered, a medium such as the Internet is extremely powerful to (anonymously) spread the sensitive information and it is shown that stopping the spread of such highly sensitive information is difficult. Therefore, program obfuscation recently attracted a lot of attention as a low cost approach to protect the inner workings of an application. However, when a new obfuscating transformation is proposed, it is unclear how to measure the quality of such transformation as there is no general agreement on this matter in this young domain. Collberg’s taxonomy [37] describes the quality of an obfuscating transformation in terms of cost, resilience and potency. The cost describes the execution penalty, the resilience measures how well a transformation withstands an attack while the potency measures how much more difficult the obfuscated code is to understand. Our work contributes by describing attacks that test the resilience of an obfuscating transformation and by the construction of a framework based on software complexity metrics to evaluate the potency of obfuscating transformations. In this dissertation, we bring together existing control flow obfuscating transformations and existing software complexity metrics. In particular, we consider three transformations: control flow flattening (CFF), branch procedures and opaque predicates together with two metrics: cyclomatic number and knot count. After applying the obfuscating transformations on a program, the complexity of the program increases. To measure this, our framework has to be capable of quantifying the obfuscating transformation independent of at which point in the development process the obfuscating transformation is applied. Therefore, our introduced framework works on the