Measuring the effectiveness of reliability growth testing

Assessing the effectiveness of reliability growth testing allows decisions to be made about the management of the programme; for example, decisions to allocate resources in an attempt to realize improvements in system reliability, or decisions to terminate testing when there is evidence that reliability has matured. However, such assessments are commonly based on measures of change in the average time between failures, which do not always provide informative measures of effectiveness. We argue that a more appropriate approach is to focus on the identification and realization of faults, which if removed are likely to lead to the improvement in system reliability. Therefore we introduce a model that incorporates the concerns of engineering experts about the likely faults in the initial system design with the information about faults that are realized on test. This model distinguishes between the processes of detecting and removing faults and so captures the effectiveness of test as well as the development of the system design. Using this model, the likely number of faults remaining in the system and the additional test time required to realize them can be estimated. This model was motivated by and assessed by engineers conducting growth tests for complex electronic systems. They consider our measures useful for supporting assessment of test effectiveness.