Automatic generation of test cases from Boolean specifications using the MUMCUT strategy

A recent theoretical study has proved that the MUMCUT testing strategy (1) guarantees to detect seven types of fault in Boolean specifications in irredundant disjunctive normal form, and (2) requires only a subset of the test sets that satisfy the previously proposed MAX-A and MAX-B strategies, which can detect the same types of fault. This paper complements previous work by investigating various methods for the automatic generation of test cases to satisfy the MUMCUT strategy. We evaluate these methods by using several sets of Boolean expressions, including those derived from real airborne software systems. Our results indicate that the greedy CUN and UCN methods are clearly better than others in consistently producing significantly smaller test sets, whose sizes exhibit linear correlation with the length of the Boolean expressions in irredundant disjunctive normal form. This study provides empirical evidences that the MUMCUT strategy is indeed cost-effective for detecting the faults considered in this paper.

[1]  D. Richard Kuhn.  Fault classes and error detection capability of specification-based testing , 1999, TSEM.

[2]  Steven P. Miller,et al.  Applicability of modified condition/decision coverage to software testing , 1994, Softw. Eng. J..

[3]  Roger E. Kirk,et al.  Statistics: An Introduction , 1998 .

[4]  Nancy G. Leveson,et al.  Requirements Specification for Process-Control Systems , 1994, IEEE Trans. Software Eng..

[5]  Tsong Yueh Chen,et al.  Using the incremental approach to generate test sets: a case study , 2003, Third International Conference on Quality Software, 2003. Proceedings..

[6]  K. A. Foster,et al.  Sensitive test data for logic expressions , 1984, SOEN.

[7]  Tsong Yueh Chen,et al.  On the Relationship Between Partition and Random Testing , 1994, IEEE Trans. Software Eng..

[8]  Bogdan Korel,et al.  Automated Software Test Data Generation , 1990, IEEE Trans. Software Eng..

[9]  Tatsuhiro Tsuchiya,et al.  On fault classes and error detection capability of specification-based testing , 2002, TSEM.

[10]  Roy P. Pargas,et al.  Test‐data generation using genetic algorithms , 1999 .

[11]  Kenneth A. Foster.  Error Sensitive Test Cases Analysis (ESTCA) , 1980, IEEE Transactions on Software Engineering.

[12]  Elaine J. Weyuker,et al.  Automatically Generating Test Data from a Boolean Specification , 1994, IEEE Trans. Software Eng..

[13]  Nancy G. Leveson,et al.  An empirical evaluation of the MC/DC coverage criterion on the HETE-2 satellite software , 2000, 19th DASC. 19th Digital Avionics Systems Conference. Proceedings (Cat. No.00CH37126).

[14]  A. Jefferson Offutt,et al.  Constraint-Based Automatic Test Data Generation , 1991, IEEE Trans. Software Eng..

[15]  Tatsuhiro Tsuchiya,et al.  Non-specification-based approaches to logic testing for software , 2002, Inf. Softw. Technol..

[16]  John Joseph Chilenski,et al.  An Investigation of Three Forms of the Modified Condition Decision Coverage (MCDC) Criterion , 2001 .

[17]  Tsong Yueh Chen,et al.  Test case selection strategies based on Boolean specifications ‡ , 2001, Softw. Test. Verification Reliab..

[18]  Tsong Yueh Chen,et al.  Two test data selection strategies towards testing of Boolean specifications , 1997, Proceedings Twenty-First Annual International Computer Software and Applications Conference (COMPSAC'97).

[19]  Willard Van Orman Quine,et al.  The Problem of Simplifying Truth Functions , 1952 .

[20]  Tsong Yueh Chen,et al.  A new heuristic for test suite reduction , 1998, Inf. Softw. Technol..

[21]  Alfred V. Aho,et al.  The Design and Analysis of Computer Algorithms , 1974 .

[22]  Man F. Lau,et al.  An extended fault class hierarchy for specification-based testing , 2005, TSEM.

[23]  P. Black,et al.  Mutation of model checker specifications for test generation and evaluation , 2001 .

[24]  Kuo-Chung Tai,et al.  Theory of Fault-Based Predicate Testing for Computer Programs , 1996, IEEE Trans. Software Eng..

[25]  Amit Paradkar,et al.  Specification‐based testing using cause‐effect graphs , 1997, Ann. Softw. Eng..

[26]  Robert M. Hierons,et al.  Testing from a Z Specification , 1997, Softw. Test. Verification Reliab..

[27]  Shaoying Liu,et al.  Generating test data from state‐based specifications , 2003, Softw. Test. Verification Reliab..

[28]  Vadim Okun,et al.  Comparison of fault classes in specification-based testing , 2004, Inf. Softw. Technol..

[29]  Yuen-Tak Yu,et al.  A comparison of MC/DC, MUMCUT and several other coverage criteria for logical decisions , 2006, J. Syst. Softw..

[30]  Kuo-Chung Tai,et al.  Test generation for Boolean expressions , 1995, Proceedings of Sixth International Symposium on Software Reliability Engineering. ISSRE'95.

[31]  Tsong Yueh Chen,et al.  MUMCUT: a fault-based strategy for testing Boolean specifications , 1999, Proceedings Sixth Asia Pacific Software Engineering Conference (ASPEC'99) (Cat. No.PR00509).

[32]  Tsong Yueh Chen,et al.  Dividing Strategies for the Optimization of a Test Suite , 1996, Inf. Process. Lett..