Benchmarking Untrustworthiness: An Alternative to Security Measurement

Benchmarking security is hard and, although there are many proposals of security metrics in the literature, no consensual quantitative security metric has been previously proposed. A key difficulty is that security is usually more influenced by what is unknown about a system than by what is known. In this paper, the authors propose the use of an untrustworthiness metric for benchmarking security. This metric, based on the idea of quantifying and exposing the trustworthiness relationship between a system and its owner, represents a powerful alternative to traditional security metrics. As an example, the authors propose a benchmark for Database Management Systems (DBMS) that can be easily used to assess and compare alternative database configurations based on minimum untrustworthiness, which is a low-cost and high-reward trust-based metric. The practical application of the benchmark in four real large database installations shows that untrustworthiness is a powerful metric for administrators to make informed security decisions by taking into account the specifics needs and characteristics of the environment being managed. DOI: 10.4018/jdtis.2010040102 International Journal of Dependable and Trustworthy Information Systems, 1(2), 32-54, April-June 2010 33 Copyright © 2010, IGI Global. Copying or distributing in print or electronic forms without written permission of IGI Global is prohibited. consider complex environments where security vulnerabilities may exist due to the combination of several distinct characteristics of the system, including the environment around it and how it is used (e.g., a database accessed by several applications and users). Insecurity metrics based on risk (Jelen & Williams, 1998) try to cope with the uncertainty associated with security goals by incorporating the probability of attacks. Risk is usually defined as the product of the likelihood of an attack by the damage expected if it happens. This metric can be used to decide if the risks are acceptable and to decide which ones have to be mitigated first. The problem is that it is very easy to underestimate or overestimate these values. This is, obviously, a major problem when they are used for supporting security related decisions. Traditional security and insecurity metrics are hard to define and compute (Torgerson, 2007) because they involve making isolated estimations about the ability of an unknown individual (e.g., a hacker) to discover and maliciously exploit an unknown system characteristic (e.g., vulnerability). In practice, it is assumed that such metrics can be computed using information about the system itself, and they depend only on the systems properties. Therefore, they are universal and have the same value when seen from different perspectives (e.g., the administrators’ versus the attackers’ point of view). In spite of the usefulness of such metrics, they are not necessarily the only way of quantifying security aspects. Consider the definition of a useful security metric: “the degree to which security goals are met in a given system allowing an administrator to make informed decisions”. An interesting alternative would be a metric that systematizes and summarizes the knowledge and control that a particular administrator has about his own system. This metric would still fit the security metric definition. Basically, the idea is not to measure just the system characteristics, but to extend the measurement to the relationship between the system and the person (or persons) that is in charge of it (defined here as the system administrator). Such a metric would allow the administrator to become aware of the security characteristics of the system, gathering knowledge to backup decisions. This metric would be even more useful for administrators that are not security experts and have to manage a complex environment, with just too many distinct security aspects to consider at once. This kind of metric is what we call a trust-based metric, in the sense that it exposes and quantifies the trustworthiness relationship between an administrator and the system he manages. In this work we argue that a highly useful trust-based metric can be based on the evaluation of how much active effort the administrator puts in his system to make it more secure. Note that effort is used broadly, including not only real effort (e.g., testing an application) but also effort put on becoming aware of the state of the system (e.g., identifying that the server currently loads insecure processes). This effort can be summarized as the level of trust (or rather distrust) that can be justifiably put in a given system as not being susceptible to attacks. As an instantiation, we propose a trust-based metric called minimum untrustworthiness that expresses the minimum level of distrust one should put in a given system or component to act accordingly to its specification. A benchmark is a procedure that allows assessing and comparing systems (or components) according to a given characteristic (e.g., performance, availability, security) (TPC, 2010). The concept of benchmarking can be summarized in three words: representativeness, usefulness, and agreement. A benchmark must be as representative as possible of a given domain but, as an abstraction of that domain, it will always be an imperfect representation of reality. However, it is useful, in the sense that its results allow making informed decisions regarding the benchmarked targets. One expected usage of a security benchmark is to compare the security characteristics of alternative systems and installations. It is in fact an invaluable tool to help administrators to become aware of the security characteristics and issues of the environments they manage. At the same time, users must agree that the benchmark 21 more pages are available in the full version of this document, which may be purchased using the "Add to Cart" button on the product's webpage: www.igi-global.com/article/benchmarking-untrustworthinessalternative-security-measurement/46937?camid=4v1 This title is available in InfoSci-Journals, InfoSci-Journal Disciplines Computer Science, Security, and Information Technology. Recommend this product to your librarian: www.igi-global.com/e-resources/libraryrecommendation/?id=2