STRAM: Measuring the Trustworthiness of Computer-Based Systems

Various system metrics have been proposed for measuring the quality of computer-based systems, such as dependability and security metrics for estimating their performance and security characteristics. As computer-based systems grow in complexity with many subsystems or components, measuring their quality in multiple dimensions is a challenging task. In this work, we tackle the problem of measuring the quality of computer-based systems based on the four key attributes of trustworthiness we developed: security, trust, resilience, and agility. In addition to conducting a systematic survey on metrics, measurements, attributes of metrics, and associated ontologies, we propose a system-level trustworthiness metric framework that accommodates four submetrics, called STRAM ( S ecurity, T rust, R esilience, and A gility M etrics). The proposed STRAM framework offers a hierarchical ontology structure where each submetric is defined as a sub-ontology. Moreover, this work proposes developing and incorporating metrics describing key assessment tools, including vulnerability assessment, risk assessment, and red teaming, to provide additional evidence in the measurement and quality of trustworthy systems. We further discuss how assessment tools are related to measuring the quality of computer-based systems and the limitations of the state-of-the-art metrics and measurements. Finally, we suggest future research directions for system-level metrics research toward measuring fundamental attributes of the quality of computer-based systems and improving the current metric and measurement methodologies.