Exploit Programming: From Buffer Overflows to "Weird Machines" and Theory of Computation

13 Hacker-driven exploitation research has developed into a discipline of its own, concerned with practical exploration of how unexpected computational properties arise in actual multi-layered, multi-component computing systems, and of what these systems could and could not compute as a result. The staple of this research is describing unexpected (and unexpectedly powerful) computational models inside targeted systems, which turn a part of the target into a so-called " weird machine " programmable by the attacker via crafted inputs (a .k .a. " exploits "). Exploits came to be understood and written as programs for these " weird machines " and served as constructive proofs that a computation considered impossible could actually be performed by the targeted environment. This research defined and fulfilled the need of such practical exploration in real systems that we must trust. Hacker research has also dominated this area, while academic analysis of the relevant computational phenomena lagged behind. We show that at its current sophistication and complexity, exploitation research as a discipline has come full circle to the fundamental questions of computability and language theory. Moreover, application of language-theoretic and computation-theoretic methods in it has already borne impressive results, helping to discover and redefine computational models and weaknesses previously overlooked. We believe it is time to bring the hacker craft of finding and programming " weird machines " inside targets and the theorists' understanding of computational models together for the next step in designing secure, trustworthy computing systems. In memory of Len Sassaman, who articulated many of the following observations, connecting the mundane and the deeply theoretical aspects of hacking. state-of-the-art hacking as a distinct research and engineering discipline that, although not yet recognized as such, harbors deep insights into the nature of computing. He has a Phd in mathematics from Northeastern University and worked at BBN Technologies on natural language processing research before coming to dartmouth. assistant professor in the Computer Science department at the University of Calgary. He seeks to understand why it seems difficult to build secure, trustworthy systems and how we can get better at it. He graduated magna cum laude from The College of New Jersey (TCNJ) with a BSc degree in computer science. dr. Locasto also holds an MSc and Phd from Columbia University. locasto@ucalgary.ca Meredith L. Patterson is a software engineer at Red Lambda. She developed the first language-theoretic defense against SQL injection in 2005, as a Phd student …