A coverage analysis tool for the effectiveness of software testing

We describe a software testing and analysis tool, called ATAC (Automatic Test Analysis for C), which is developed as a research instrument at Bellcore to measure the effectiveness of testing data. The design, functionality, and usage of ATAC are presented in this paper. Furthermore, to demonstrate the capability and applicability of ATAC, we obtain the 12 program versions of a critical industrial application developed in a recent university/industry N-Version Software project, and use the ATAC tool to analyze and compare coverage of the testing conducted in the program versions. Preliminary results from this investigation show that ATAC could be a powerful testing tool to provide testing metrics and quality control guidance for the certification of high quality software components or systems. It can also assist software reliability researchers and practitioners in searching for the missing link between structure-based testing schemes and software reliability.

[1]  Darrel C. Ince,et al.  The Automatic Generation of Test Data , 1987, Comput. J..

[2]  J. R. Horgan,et al.  A data flow coverage testing tool for C , 1992, [1992] Proceedings of the Second Symposium on Assessment of Quality Software Development Tools.

[3]  Murray Hill,et al.  Yacc: Yet Another Compiler-Compiler , 1978 .

[4]  Vi Andrew Jefferson Offutt,et al.  Automatic test data generation , 1988 .

[5]  Michael R. Lyu,et al.  Applying reliability models more effectively (software) , 1992, IEEE Software.

[6]  Algirdas Avizienis,et al.  The N-Version Approach to Fault-Tolerant Software , 1985, IEEE Transactions on Software Engineering.

[7]  Michael R. Lyu,et al.  Improving the N-version programming process through the evolution of a design paradigm , 1993 .

[8]  John F. Passafiume,et al.  Software testing and evaluation , 1987 .

[9]  Peter M. Maurer,et al.  Generating test data with enhanced context-free grammars , 1990, IEEE Software.

[10]  R.A. DeMillo,et al.  An extended overview of the Mothra software testing environment , 1988, [1988] Proceedings. Second Workshop on Software Testing, Verification, and Analysis.

[11]  Elaine J. Weyuker,et al.  An Applicable Family of Data Flow Testing Criteria , 1988, IEEE Trans. Software Eng..

[12]  Michael R. Lyu,et al.  Assuring Design Diversity in N-Version Software: A Design Paradigm for N-Version Programming , 1992 .

[13]  William E. Howden,et al.  Functional program testing and analysis , 1986 .

[14]  Elaine J. Weyuker,et al.  Selecting Software Test Data Using Data Flow Information , 1985, IEEE Transactions on Software Engineering.

[15]  C. Mallows,et al.  When Should One Stop Testing Software , 1988 .