Mapping the Field of Software Security Metrics

While security, or its absence, is a property of running software, many aspects of software requirements, design, implementation, and testing contribute to the presence or absence of security in the finished product. Assessing whether a given piece of software meets a set of security objectives is a multi-dimensional problem, and we do not yet have a clear picture of all of the dimensions. The goal of this research is to support researcher and practitioner use of security measurement by cataloging available metrics, their validation, and the subjects they measure through conducting a systematic mapping study. Our study began with 1,561 papers and narrowed down to 63 papers reporting on 346 metrics. For each metric, we identify the subject being measured, how the metric has been evaluated by researcher(s), and how the metric is being used. Approximately 85% of security-specific metrics have been proposed and evaluated solely by their authors. Approximately 40% of the metrics are not empirically evaluated, and many artifacts and processes remain unmeasured. Approximately 15% of the metrics focus on the early stages of development or on testing (1.5%). At present, despite the abundance of metrics found in the literature, those available give us an incomplete, disjointed, hazy view of software security.