A criticism on the capture-and-recapture method for software reliability assurance

There is a great deal of literature claiming that the capture-and-recapture (CR) method can estimate the number of remaining software bugs during the test process, and this enables us to decide when to stop testing. Although the decision about when to stop testing can only be made when there is ample confidence that the number of remaining bugs is less than a certain limit, most of the literature just makes a point estimate of it, using a simple proportional formula, without considering the level of errors. Some of the literature touches on the fact that the recaptured bug number is given by a hypergeometric distribution, but none uses the confidence interval estimate effectively. This paper derives a formula giving a confidence interval estimate for the seeded bug ratio. Then it presents a virtual experiment to apply the CR method. The paper concludes that the CR method cannot be effectively applied to software testing because it overestimates the number of bugs to be detected before testing can be stopped, and the CR method is not applicable at all to the final phase of testing because it is very difficult to make enough seeded bugs that are equivalent to those remaining in the target software.