Towards Automatic Compartmentalization of C Programs on Capability Machines

Capability-based protection mechanisms can offer fine-grained memory protection (through memory capabilities), as well as fine-grained protection of general software-defined objects (through object capabilities). Because of the similarity that capabilities have to the notion of pointer in C, compilers can use the capability mechanisms offered by the target platform to generate code that is more resilient to attack. For instance, C arrays can be compiled to memory capabilities thus providing hardwareenforced spatial safety guarantees and hence strong resilience against buffer overflow attacks. State-of-the-art capability-based systems (like for instance the CHERI system [1]) come with a C compiler that provides such safety guarantees. But such safe compilation does not provide security guarantees for an attacker model where an attacker can compromise part of the code of an application, for instance by providing a malicious library, possibly in compiled form. An application is still executed in a single protection domain. The mechanism of object capabilities can be used to remedy this: object capabilities support compartmentalization of an application where different parts of the application can be executed in different protection domains, and hence one part of the application can be protected against malicious behaviour in other parts. However, to the best of our knowledge, state-of-the-art C compilers provide no automatic support for such compartmentalization. In CHERI, support for such compartmentalization is offered as an API [2]. This paper reports on our work-in-progress on the definition, implementation and evaluation of a compiler that automatically compartmentalizes the programs it compiles, essentially by executing each C compilation unit in a separate protection domain. We provide a formal definition of our compiler, and an implementation on CHERI as a source-to-source compiler that can detect and insert the necessary invocations to CHERI’s API for compartmentalization. We illustrate the security properties of the compiler by means of examples and discuss our work-inprogress on formalizing and proving these security properties. This paper uses colours to distinguish elements of different languages. For a better experience, please print/view it in colour.