What we have learned about the value of software assurance

Context: There is a pervasive feeling that somehow software assurance is important. For example, national space agencies, such as the National Aeronautics and Space Administration (NASA) and the Japan Aerospace Exploration Agency (JAXA), require assurance for their critical software systems. Their assurance effort goes beyond testing, and includes activities such as process compliance checks, artifact audits, and traceability validation. These activities can be costly and their benefits, particularly in managing risk, are poorly understood. This inevitably leads to tough questions concerning value such as "Is assurance worth doing?" and "How much assurance do we actually need?" Goal: For many years the software assurance community has been struggling with how to rationalize investing in software assurance. Numerous value models have been suggested, but these have not been able to explicitly connect assurance activities to their expected benefits, making them difficult to use for value assessment. As a consequence, assurance managers are ill equipped to plan and justify their budgets, frequently finding that assurance is first on the chopping block when cuts are made. We discuss some common value propositions and how why they are not operationally useful within our assurance practice. Method: From our ongoing empirical study of assurance practice at NASA and JAXA, we are led to a fundamental assurance value proposition that implies an operational and justifiable proposition of assurance value. We depart from contemporary views that focus on defect-centric factors such as defect avoidance or early defect mitigation. Rather, we have observed that assurance value stems from enabling more confident quality-critical decision-making. Results: From this new viewpoint value can be tangibly measured as a reduction in the risk of making a bad decision due to uncertainty in quality factors. This has some surprising and consequential implications. For example, from this viewpoint, assurance value does not depend on the actual number of defects found; instead it depends on the degree of coverage an assurance activity provides, regardless if defects are found. Conclusions: By exploring what we have learned about assurance value we better understand and from an industry standpoint it is an important issue to address.