Scalable security architecture for trusted software

Security-critical tasks executing on general-purpose computers require protection against software and hardware attacks to achieve their security objectives. Security services providing this protection can be offered by mechanisms rooted in processor hardware, since its storage and computing elements are typically outside the reach of attackers. This thesis presents the Bastion architecture, a hardware-software security architecture for providing protection scalable to a large number of security-critical tasks. Protection is enabled by three sets of new mechanisms: for protecting a trusted hypervisor, for fine-grained protection of modules in application or operating system space, and for securing the input and output of Bastion-protected software modules. This thesis also presents an implementation and evaluation of Bastion, and explores alternatives for one of its core security functions: memory authentication. The hypervisor, a layer of software dedicated to the virtualization of machine resources, is increasingly being involved in security solutions. We use it in Bastion as a manager of security-critical tasks. While past solutions protect the hypervisor from runtime software attacks, Bastion also protects the hypervisor from physical attacks, protects it from offline attacks, and provides it with a secure launch mechanism. Within this protected Bastion hypervisor, we design a second set of mechanisms that provide separate execution compartments for each security-critical task running in the virtual machines hosted by the hypervisor. These compartments are protected against both hardware attacks and software attacks originating from a potentially compromised operating system. To enable security-critical tasks to communicate with the outside world, we provide a third set of mechanisms for secure input and output to and from Bastion-protected compartments. We implement and evaluate a Bastion prototype by modifying the source code of the OpenSPARC processor and hypervisor systems. Addionally, we survey the design space of alternatives to the Bastion memory authentication mechanism, which is central to protecting critical software execution in Bastion. These contributions can improve security in the digital world by informing the design of the next generation of general-purpose computing platforms.