Security Applications of Formal Language Theory

We present a formal language theory approach to improving the security aspects of protocol design and message-based interactions in complex composed systems. We argue that these aspects are responsible for a large share of modern computing systems' insecurity. We show how our approach leads to advances in input validation, security modeling, attack surface reduction, and ultimately, software design and programming methodology. We cite examples based on real-world security flaws in common protocols, representing different classes of protocol complexity. We also introduce a formalization of an exploit development technique, the parse tree differential attack, made possible by our conception of the role of formal grammars in security. We also discuss the negative impact unnecessarily increased protocol complexity has on security. This paper provides a foundation for designing verifiable critical implementation components with considerably less burden to developers than is offered by the current state of the art. In addition, it offers a rich basis for further exploration in the areas of offensive analysis and, conversely, automated defense tools, and techniques.