The relationship between test coverage and reliability

Models the relationship between testing effort, coverage and reliability, and presents a logarithmic model that relates testing effort to test coverage: statement (or block) coverage, branch (or decision) coverage, computation use (c-use) coverage, or predicate use (p-use) coverage. The model is based on the hypothesis that the enumerables (like branches or blocks) for any coverage measure have different detectability, just like the individual defects. This model allows us to relate a test coverage measure directly to the defect coverage. Data sets for programs with real defects are used to validate the model. The results are consistent with the known inclusion relationships among block, branch and p-use coverage measures. We show how the defect density controls the time-to-next-failure. The model can eliminate variables like the test application strategy from consideration. It is suitable for high-reliability applications where automatic (or manual) test generation is used to cover enemerables which have not yet been tested.<<ETX>>

[1]  Elaine J. Weyuker,et al.  More Experience with Data Flow Testing , 1993, IEEE Trans. Software Eng..

[2]  Vernon Rego,et al.  A Time/Structure Based Model for Estimating Software Reliability , 1992 .

[3]  Janet R. Dunham Experiments in software reliability: Life-critical applications , 1986, IEEE Transactions on Software Engineering.

[4]  Robert B. Grady,et al.  Practical Software Metrics for Project Management and Process Improvement , 1992 .

[5]  Elaine J. Weyuker,et al.  An Applicable Family of Data Flow Testing Criteria , 1988, IEEE Trans. Software Eng..

[6]  Simeon C. Ntafos,et al.  A Comparison of Some Structural Testing Strategies , 1988, IEEE Trans. Software Eng..

[7]  Pradip K. Srimani,et al.  The nature of fault exposure ratio , 1992, [1992] Proceedings Third International Symposium on Software Reliability Engineering.

[8]  Paul Piwowarski,et al.  Coverage measurement experience during function test , 1993, Proceedings of 1993 15th International Conference on Software Engineering.

[9]  Phyllis G. Frankl,et al.  An Experimental Comparison of the Effectiveness of Branch Testing and Data Flow Testing , 1993, IEEE Trans. Software Eng..

[10]  Yashwant K. Malaiya,et al.  The Coverage Problem for Random Testing , 1984, ITC.

[11]  Yashwant K. Malaiya,et al.  Predictability of software-reliability models , 1992 .

[12]  Victor R. Basili,et al.  Analyzing the test process using structural coverage , 1985, ICSE '85.

[13]  M. Trachtenberg Why failure rates observe Zipf's law in operational software , 1992 .

[14]  Michael R. Lyu,et al.  A coverage analysis tool for the effectiveness of software testing , 1994 .

[15]  Jeffrey M. Voas,et al.  Improving the software development process using testability research , 1992, [1992] Proceedings Third International Symposium on Software Reliability Engineering.

[16]  Boris Beizer,et al.  Software Testing Techniques , 1983 .

[17]  Pradip K. Srimani,et al.  An Examination of Fault Exposure Ratio , 1993, IEEE Trans. Software Eng..

[18]  Michael R. Lyu,et al.  What is software reliability? , 1994, Proceedings of COMPASS'94 - 1994 IEEE 9th Annual Conference on Computer Assurance.

[19]  Ann Marie Neufelder,et al.  Ensuring Software Reliability , 1992 .

[20]  Edward J. McCluskey,et al.  Pseudorandom Testing , 1987, IEEE Transactions on Computers.

[21]  Vishwani D. Agrawal,et al.  A Statistical Theory of Digital Circuit Testability , 1990, IEEE Trans. Computers.

[22]  A. Jefferson Offutt,et al.  An Empirical Evaluation , 1994 .

[23]  E. J. Weyuker An empirical study of the complexity of data flow testing , 1988, [1988] Proceedings. Second Workshop on Software Testing, Verification, and Analysis.

[24]  H. Hecht,et al.  Rare conditions and their effect on software failures , 1994, Proceedings of Annual Reliability and Maintainability Symposium (RAMS).

[25]  James M. Bieman,et al.  An empirical evaluation (and specification) of the all-du-paths testing criterion , 1992, Softw. Eng. J..

[26]  Joseph Robert Horgan,et al.  Reliable software and communication: software quality, reliability, and safety , 1993, Proceedings of 1993 15th International Conference on Software Engineering.

[27]  James M. Bieman,et al.  Estimating the number of test cases required to satisfy the all-du-paths testing criterion , 1989, TAV3.

[28]  Yashwant K. Malaiya,et al.  Enhancing accuracy of software reliability prediction , 1993, Proceedings of 1993 IEEE International Symposium on Software Reliability Engineering.