Measurement and enhancement of software reliability through testing

Any software must be sufficiently reliable to be useful. To produce reliable software products on schedule and within budget, software quality assurance engineers need to measure and predict software reliability from time to time. Software reliability models allow this measurement and prediction. Software testing is the major approach for achieving high reliability. Some researchers proposed to test programs based their usage and claimed that usage based testing is 20 times more effective than coverage testing. We find that the claim is incorrect and that the optimal test input distribution depends not only on the operational profile, but also on the defect detectability profile and the planned amount of testing. For limited testing, input distribution should be more biased than the operational profile. For extensive testing, input should be more uniformly distributed. We model the relation between fault exposure ratio and static software metrics to allow early prediction of the software reliability growth. Several empirical techniques for enhancing the predictive quality of SRGMs are introduced and evaluated, including: noise smoothing through fixed-size grouping, lump grouping, windowing and model bias adjustment through weighted least squares and recalibration. The results show that proper use of such enhancing techniques can significantly improve the predictive accuracy of SRGMs. Coverage based reliability modeling takes into account the differences in test effectiveness. We propose a logarithmic model for test coverage and relate defect coverage to test coverage with a three parameter model. Existing software reliability tools lag too much behind the research in the field. We propose a framework for linking the isolated research results together and present a tool incorporating various existing modeling techniques. The new tool named ROBUST employs knowledge of static software complexity metrics, dynamic failure data and various test coverages to support software reliability estimation and prediction at different software development phases.