Reverse analysis of secure communication protocol based on taint analysis

To maintain communications confidentiality, security protocols are widely used in more and more network applications. Moreover, some malwares even leverage these kinds of protocols to evade inspection by IDS. Most security protocols are designed and verified by formalized methods; however, observation shows that protocol implementations commonly contain flaws or vulnerabilities. Therefore, research on reverse engineering of security protocols can play an important role in improving the security of network applications, especially by providing another way to fight against malwares. Nevertheless, previous protocol reverse engineering technologies, which are based on analysis of network traces, encounter great challenges when the network messages transmitted between different protocol principals are encrypted. This paper proposes a taint analysis based method, which aims to infer the message format from dynamic execution of security protocol applications. The proposed approach is based on the observation that the process of message parsing in cryptographic protocol applications reveals rich information about the hierarchical structures and semantics of their messages. Hence, by observing calls to library function and instruction execution in network programs, the proposed approach can reverse derive large amount of information about their protocol, such as message format and protocol model, even the communication is encrypted. Experiments show that the reverse analysis results not only accurately identify message fields, but also unveil the structure of the encrypted message fields.