Experimentation Support for Cyber Security Evaluations

To improve the information assurance of mission execution over modern IT infrastructure, new cyber defenses need to not only provide security benefits, but also perform within a given cost regime. Current approaches for validating and integrating cyber defenses heavily rely on manual trial-and-error, without a clear and systematic understanding of security versus cost tradeoffs. Recent work on model-based analysis of cyber defenses has led to quantitative measures of the attack surface of a distributed system hosting mission critical applications. These metrics show great promise, but the cost of manually creating the underlying models is an impediment to their wider adoption. This paper describes an experimentation framework for automating multiple activities associated with model construction and validation, including creating ontological system models from real systems, measuring and recording distributions of resource impact and end-to-end performance overhead values, executing real attacks to validate theoretic attack vectors found through analytic reasoning, and creating and managing multi-variable experiments.