A cost-benefit stopping criterion for statistical testing

Determining when to stop a statistical test is an important management decision. Several stopping criteria have been proposed, including criteria based on statistical similarity, the probability that the system has a desired reliability, and the expected cost of remaining faults. This paper proposes a new stopping criterion based on a cost-benefit analysis using the expected reliability of the system (as opposed to an estimate of the remaining faults). The expected reliability is used, along with other factors such as units deployed and expected use, to anticipate the number of failures in the field and the resulting anticipated cost of failures. Reductions in this number generated by increasing the reliability are balanced against the cost of further testing to determine when testing should be stopped.

[1]  Robert Oshana,et al.  Improving software quality using statistical testing techniques , 2000, Inf. Softw. Technol..

[2]  David Wright,et al.  Some Conservative Stopping Rules for the Operational Testing of Safety-Critical Software , 1997, IEEE Trans. Software Eng..

[3]  Jesse H. Poore,et al.  Improved techniques for software testing based on markov chain usage models , 1999 .

[4]  Jesse H. Poore,et al.  Stopping criteria for statistical testing , 2000, Inf. Softw. Technol..

[5]  C. Mallows,et al.  When Should One Stop Testing Software , 1988 .

[6]  Shaoying Liu,et al.  Criteria for generating specification-based tests , 1999, Proceedings Fifth IEEE International Conference on Engineering of Complex Computer Systems (ICECCS'99) (Cat. No.PR00434).

[7]  James A. Whittaker,et al.  A Markov Chain Model for Statistical Software Testing , 1994, IEEE Trans. Software Eng..

[8]  Jesse H. Poore,et al.  Planning and certifying software system reliability , 1993, IEEE Software.

[9]  Ming Zhao,et al.  Planning Models for Software Reliability and Cost , 1998, IEEE Trans. Software Eng..

[10]  Amrit L. Goel,et al.  Software Reliability Models: Assumptions, Limitations, and Applicability , 1985, IEEE Transactions on Software Engineering.

[11]  Shirley Mills,et al.  A Binary Markov Process Model for Random Testing , 1996, IEEE Trans. Software Eng..

[12]  Swapna S. Gokhale,et al.  A non-homogeneous Markov software reliability model with imperfect repair , 1996, Proceedings of IEEE International Computer Performance and Dependability Symposium.

[13]  Jesse H. Poore,et al.  Statistical testing of software based on a usage model , 1995, Softw. Pract. Exp..

[14]  Anneliese Amschler Andrews,et al.  High quality behavioral verification using statistical stopping criteria , 2001, Proceedings Design, Automation and Test in Europe. Conference and Exhibition 2001.

[15]  Jeffrey M. Voas,et al.  Estimating the Probability of Failure When Testing Reveals No Failures , 1992, IEEE Trans. Software Eng..

[16]  Tom Chávez,et al.  A Decision-Analytic Stopping Rule for Validation of Commercial Software Systems , 2000, IEEE Trans. Software Eng..

[17]  P. Sassone,et al.  Cost-Benefit Analysis: A Handbook , 1978 .

[18]  Harvey S. Koch,et al.  Optimal Release Time of Computer Software , 1983, IEEE Transactions on Software Engineering.

[19]  Amrit L. Goel,et al.  Optimum release time for software systems based on reliability and cost criteria , 1984, J. Syst. Softw..

[20]  Amrit L. Goel,et al.  Time-Dependent Error-Detection Rate Model for Software Reliability and Other Performance Measures , 1979, IEEE Transactions on Reliability.