Identifying the mechanisms driving code inspection costs and benefits

Software inspections have long been considered to be an effective way to detect and remove defects from software. However, there are costs associated with carrying out inspections and these costs may outweigh the expected benefits. It is important to understand the tradeoffs between these costs and benefits. We believe that these are driven by several mechanisms, both internal and external to the inspection process. Internal factors are associated with the manner in which the steps of the inspection are organized into a process (structure), as well as the manner in which each step is carried out (technique). External ones include differences in reviewer ability and code quality (inputs), and interactions with other inspections, the project schedule, personal calendars, etc. (environment). We started a study to identify the mechanisms that strongly influence an inspection's costs and effectiveness. Most of the existing literature on inspections have discussed how to get the most benefit out of inspections by proposing changes to the process structure, but with little or no empirical work conducted to demonstrate how they worked better and at what cost. We hypothesized that these changes will affect the defect detection effectiveness of the inspection, but that any increase in effectiveness will have a corresponding increase in inspection interval and effort. We evaluated this hypothesis with a controlled experiment on a live development project at Lucent Technologies, using professional software developers. We found that these structural changes were largely ineffective in improving the effectiveness of inspections, but certain treatments dramatically increased the inspection interval. We also noted a large amount of unexplained variance in the data suggesting that other factors must have a strong influence on inspection performance. On further investigation, we found that the inputs into the process (reviewers and code units) account for more of the variation than the original treatment variables, leading us to conclude that better techniques by which reviewers detect defects, not better process structures, are the key to improving inspection effectiveness.