Code Obfuscation and Malware Detection by Abstract Interpretation

Functions We already observed in Section 5.2.2 that a function f : Z → Z is decomposed into elementary functions, i.e., assembly instructions within some basic block. Following the same approach, let us assume that the function f can be expressed as a composition of elementary functions, namely f = λx.h(g1(x, ..., x), ..., gk(x, ..., x)) where h : Zk → Z and gi : Zni → Z. More in general, each gi can be further decomposed into elementary functions. For example, f(x) = x2 + x is decomposed as h(g1(x), g2(x)) where h(x, y) = x+ y, g1(x) = x 2 and g2(x) = x. Let us consider the pointwise extensions of the elementary functions, which are still denoted, with a slight abuse of notation, by h : ℘(Z)k → ℘(Z) and gi : ℘(Z)ni → ℘(Z), and let us denote their composition by F def = λX.h(g1(X, ...,X), ..., gk (X, ...,X)) : ℘(Z) → ℘(Z) For example, for the above decomposition f(x) = x2 + x = h(g1(x), g2(x)), we have that F : ℘(Z) → ℘(Z) is as follows: F (X) = {y2 + z | y, z ∈ X}. Observe that F does not coincide with the pointwise extension f of f , e.g., F ({1, 2}) = {2, 3, 5, 6} while f({1, 2}) = {2, 6}. Let us also notice that F on singletons coincides with f , namely for any x ∈ Z, F ({x}) = f(x). Thus, the concrete test CT can be equivalently formulated as ∀x ∈ Z : F ({x}) ⊆ nZ. Let A ∈ uco(℘(Z)) be an abstract domain such that there exists some an ∈ A with γA(an) = nZ. The attacker A approximates the computation of function F : ℘(Z) → ℘(Z) in a step by step fashion, meaning that A approximates every elementary function composing F . Thus, the abstract function F ♯ : A → A is defined as the composition of the best correct approximations h and g i on A of the elementary functions, namely: F (a) def = αA(h(γA(αA(g1(γA(a), ..., γA(a)))), ..., γA(αA(gk(γA(a), ..., γA(a)))))) = h(g i (a), ..., g A k (a)) 106 5 Control Code Obfuscation When the abstract test AT ♯ A for F ♯ on A holds, the attacker modeled by the abstract domain A classifies the predicate n|f(x) as opaque. It turns out that F ♯ is a correct approximation of F on A, namely αA ◦ F ⊑A F ♯ ◦ αA, and this guarantees the soundness of the abstract test AT ♯ A . Corollary 5.8. AT ♯ A is sound. proof: We first show that F ♯ : A → A is a sound approximation of F : ℘(Z) → ℘(Z), namely ∀X ∈ ℘(Z) : αA(F (X)) ≤A F (αA(X)). In fact for any X ∈ ℘(Z): αA(F (X)) = αA(h(g1(X, ..., X), ..., gk(X, ..., X))) ≤A αA(h(γA(g1(X, ..., X), ..., γA(gk(X, ..., X))))) ≤A αA(h(γA(αA(g1(γA(αA(X)), ..., γA(αA(X))))), ..., γA(αA(gk(γA(αA(X)), ..., γA(αA(X))))))))