Stopping criteria for statistical testing

Abstract The decision to stop testing can be based on a number of criteria, such as (1) the confidence in a reliability estimate; (2) the degree to which testing experience has converged to the expected use of the software; and (3) model coverage criteria based on a degree of state, arc, or path coverage during crafted and random testing. In practice it is best to use multiple stopping criteria. For example, further evaluation of the testing performed is needed if the measure of correspondence between testing experience and expected use of the software indicates that the testing experience closely matches the expected use of the software, yet the variance of the reliability estimate is unacceptably large. One test of equality of testing experience and expected use is the Kullback discriminant from the usage chain to the testing chain. A new measure of “approximate equality” is introduced here for use in conjunction with the Kullback discriminant.