Security: Where Testing Fails

Abstract : Computer security addresses the problem of enforcement of security policies in the presence of malicious users and software. Systems enforcing mandatory policies can create confinement domains that limit the damage incurred by malicious software executing in applications. To achieve assurance that the confinement domains cannot be breached, the underlying enforcement mechanism must be constructed to ensure that it is resistant to penetration by malicious software and is free of malicious artifacts. The limitations and contributions of testing in achieving these goals are discussed. Why would a national software testing laboratory advertise on its web page that it provides testing for functionality, compatibility, performance, scalability, and fault tolerance, but not security? The answer may lie in the fact that certain aspects of security policy can be described in completely non-subjective terms. For example, the policy may state that unauthorized individuals are not permitted to read classified material. Can testing ensure that policy will not be violated? This paper provides an overview of challenges that security poses to testing and describes the role of testing in the engineering of trustworthy systems.