Software Challenges

In software, “benchmarking” usually compares two companies' practices and results, but, occasionally, it involves sets of companies. For example, there are benchmark comparisons of industry software, such as insurance software, military software, telecommunication software, commercial software, and the like. In other domains, “benchmark” usually means the collection of a substantial body of quantitative data. Benchmark comparisons of various computers, for example, rate their relative performance in at least half a dozen categories. Historically, software benchmarks have been qualitative rather than quantitative. Even the Software Engineering Institute's Capability Maturity Model (SEI CMM) is essentially a qualitative benchmark that ranks company performance on a five-point scale that lacks quantification for specific quality and productivity levels