Nist report takes a step toward better testing

A fter decades of " by guess and by gosh " estimating of just how costly inadequately tested software is to both developers and users, the National Institute of Standards and Technology has released what could be the topic's Rosetta Stone. The Economic Impacts of Inadequate Infrastructure for Software Testing, released in May, exhaustively studied the problems associated with defective software—first in the transportation manufacturing equipment industries and then in the financial services industry. In transport-related industries, the study estimates that developers and users spend US$1.8 billion annually to correct flaws resulting from inadequate testing. The study also asserts that adopting feasible testing improvements could eliminate a third of those costs. In the financial services industry , the annual cost is $3.3 billion, with a potential $1.5 billion in savings through adopting feasible improvements. For the first time, NIST officials have expanded the statistics to provide a nationwide estimate of how costly the lack of a robust testing infrastructure is; based on extrapolation of the two studied industry sectors, NIST estimates the annual national cost of inadequate testing can be as much as $59 billion. Up to $22 billion of that could be saved if licensed software had just 50 percent fewer bugs. " A study like this is somewhat unusual in terms of the depth of analysis, " says Gregory Tassey, NIST's senior economist. " We chose two large and important and quite different sectors of the economy to look at. That gave us a broad-enough perspective on software testing in general to rationalize doing an ex-trapolation to the national economic level. " Tassey cited the report's extensive primary source information. The study's transportation equipment portion surveyed 10 vendors and 179 users of computer-aided design/manufac-turing/engineering and product data management software. The financial services portion surveyed four vendors and 98 users of financial exchange software. Several questions exist, however, about the report's ultimate effect. At 300 pages—includ-ing references and an appendix—it's daunting to sift through for the nonstatistician. And, coming on the heels of several well-publicized initiatives to improve software security indus-trywide, the report could cause confusion as to the problems it addresses. Carnegie Mellon University researcher William Scherlis is also codirector of the Sustainable Computing Consortium (SCC), a new initiative meant to address several of the issues defective software has raised across the software and user communities. The consortium's formation was announced in May, almost simultaneous with the NIST …