Software Assurance Metrics and Tool Evaluation

The U.S. National Institute of Standards and Technology (NIST) is starting two ambitious projects to (1) develop a taxonomy of software security flaws and vulnerabilities, (2) develop a taxonomy of software assurance (SA) functions and techniques which detect those flaws, (3) perform and maintain a survey of SA tools implementing the functions, (4) develop testable specifications of SA functions and explicit tests, include a standard reference dataset, to evaluate how closely tools implement the functions, and (5) lead efforts to develop metrics for the effectiveness of those functions. The end result is that users will be able to choose a combination of techniques which best suits their needs and will be able to state how much confidence they have in software which has been assessed. This paper details these two projects and presents our justifications and expectations.