Software reliability growth with test coverage

"Software test-coverage measures" quantify the degree of thoroughness of testing. Tools are now available that measure test-coverage in terms of blocks, branches, computation-uses, predicate-uses, etc. that are covered. This paper models the relations among testing time, coverage, and reliability. An LE (logarithmic-exponential) model is presented that relates testing effort to test coverage (block, branch, computation-use, or predicate-use). The model is based on the hypothesis that the enumerable elements (like branches or blocks) for any coverage measure have various probabilities of being exercised; just like defects have various probabilities of being encountered. This model allows relating a test-coverage measure directly with defect-coverage. The model is fitted to 4 data-sets for programs with real defects. In the model, defect coverage can predict the time to next failure. The LE model can eliminate variables like test-application strategy from consideration. It is suitable for high reliability applications where automatic (or manual) test generation is used to cover enumerables which have not yet been tested. The data-sets used suggest the potential of the proposed model. The model is simple and easily explained, and thus can be suitable for industrial use. The LE model is based on the time-based logarithmic software-reliability growth model. It considers that: at 100% coverage for a given enumerable, all defects might not yet have been found.

[1]  Mladen A. Vouk,et al.  Using Reliability Models During Testing With Non-Operational Profiles* , 2002 .

[2]  Yashwant K. Malaiya,et al.  The Coverage Problem for Random Testing , 1984, ITC.

[3]  Yashwant K. Malaiya,et al.  Predictability of software-reliability models , 1992 .

[4]  Michael R. Lyu,et al.  An empirical study of the correlation between code coverage and reliability estimation , 1996, Proceedings of the 3rd International Software Metrics Symposium.

[5]  Michael R. Lyu,et al.  A coverage analysis tool for the effectiveness of software testing , 1993, Proceedings of 1993 IEEE International Symposium on Software Reliability Engineering.

[6]  Ralph A. Evans,et al.  IEEE transactions on reliability , 2004, IEEE Transactions on Reliability.

[7]  Joseph Robert Horgan,et al.  Reliable software and communication: software quality, reliability, and safety , 1993, Proceedings of 1993 15th International Conference on Software Engineering.

[8]  Thomas J. Ostrand,et al.  Experiments on the effectiveness of dataflow- and control-flow-based test adequacy criteria , 1994, Proceedings of 16th International Conference on Software Engineering.

[9]  James M. Bieman,et al.  The relationship between test coverage and reliability , 1994, Proceedings of 1994 IEEE International Symposium on Software Reliability Engineering.

[10]  James M. Bieman,et al.  An empirical evaluation (and specification) of the all-du-paths testing criterion , 1992, Softw. Eng. J..

[11]  Yashwant K. Malaiya,et al.  Estimating the Number of Residual Defects , 1998 .

[12]  Mitsuru Ohba,et al.  Does imperfect debugging affect software reliability growth? , 1989, ICSE '89.

[13]  Yashwant K. Malaiya,et al.  Estimating the number of residual defects [in software] , 1998, Proceedings Third IEEE International High-Assurance Systems Engineering Symposium (Cat. No.98EX231).

[14]  Yashwant K. Malaiya,et al.  What do the software reliability growth model parameters represent? , 1997, Proceedings The Eighth International Symposium on Software Reliability Engineering.

[15]  H. Hecht,et al.  Rare conditions and their effect on software failures , 1994, Proceedings of Annual Reliability and Maintainability Symposium (RAMS).

[16]  William Farr,et al.  Software reliability modeling survey , 1996 .

[17]  Elaine J. Weyuker,et al.  Selecting Software Test Data Using Data Flow Information , 1985, IEEE Transactions on Software Engineering.

[18]  Pradip K. Srimani,et al.  An Examination of Fault Exposure Ratio , 1993, IEEE Trans. Software Eng..

[19]  John D. Musa,et al.  Software reliability measurement , 1984, J. Syst. Softw..

[20]  John D. Musa,et al.  Software Reliability Engineering , 1998 .

[21]  Mladen A. Vouk Software Reliability Engineering , 1999 .

[22]  Hoang Pham Software Reliability , 1999 .

[23]  Victor R. Basili,et al.  Analyzing the test process using structural coverage , 1985, ICSE '85.

[24]  Vernon Rego,et al.  A Time/Structure Based Model for Estimating Software Reliability , 1992 .

[25]  Robert B. Grady,et al.  Practical Software Metrics for Project Management and Process Improvement , 1992 .

[26]  Lori A. Clarke,et al.  A Formal Evaluation of Data Flow Path Selection Criteria , 1989, IEEE Trans. Software Eng..

[27]  Paul Piwowarski,et al.  Coverage measurement experience during function test , 1993, Proceedings of 1993 15th International Conference on Software Engineering.

[28]  Phyllis G. Frankl,et al.  An Experimental Comparison of the Effectiveness of Branch Testing and Data Flow Testing , 1993, IEEE Trans. Software Eng..

[29]  Alberto Pasquini,et al.  Sensitivity of reliability growth models to operational profile errors , 1996, Proceedings of ISSRE '96: 7th International Symposium on Software Reliability Engineering.