A Comparison of Tabular Expression-Based Testing Strategies

Tabular expressions have been proposed as a notation to document mathematically precise but readable software specifications. One of the many roles of such documentation is to guide testers. This paper 1) explores the application of four testing strategies (the partition strategy, decision table-based testing, the basic meaningful impact strategy, and fault-based testing) to tabular expression-based specifications, and 2) compares the strategies on a mathematical basis through formal and precise definitions of the subsumption relationship. We also compare these strategies through experimental studies. These results will help researchers improve current methods and will enable testers to select appropriate testing strategies for tabular expression-based specifications.

[1]  Kathryn L. Heninger.  Specifying Software Requirements for Complex Systems: New Techniques and Their Application , 2001, IEEE Transactions on Software Engineering.

[2]  D. Richard Kuhn.  Fault classes and error detection capability of specification-based testing , 1999, TSEM.

[3]  Sigrid Eldh.  Software Testing Techniques , 2007 .

[4]  Ryszard Janicki,et al.  On a formal semantics of tabular expressions , 2001, Sci. Comput. Program..

[5]  Ryszard Janicki,et al.  Tabular Expressions and Their Relational Semantics , 2005, Fundam. Informaticae.

[6]  Emil Sekerinski,et al.  Exploring Tabular Verification and Refinement , 2003, Formal Aspects of Computing.

[7]  Glenford J. Myers,et al.  Art of Software Testing , 1979 .

[8]  David Lorge Parnas,et al.  Using Test Oracles Generated from Program Documentation , 1998, IEEE Trans. Software Eng..

[9]  David Lorge Parnas,et al.  Inspection of Safety-Critical Software Using Program-Function Tables , 2001, IFIP Congress.

[10]  Steven P. Miller,et al.  Applicability of modified condition/decision coverage to software testing , 1994, Softw. Eng. J..

[11]  Shilei Liu,et al.  Generating Test Cases from Software Documentation Title: Generating Test Cases from Software Documentation , 2001 .

[12]  Matthias Grochtmann,et al.  Classification trees for partition testing , 1993, Softw. Test. Verification Reliab..

[13]  David Lorge Parnas,et al.  Disciplined methods of software specification: a case study , 2005, International Conference on Information Technology: Coding and Computing (ITCC'05) - Volume II.

[14]  A. Jefferson Offutt,et al.  Experimental results from an automatic test case generator , 1993, TSEM.

[15]  David Lorge Parnas,et al.  Documentation of requirements for computer systems , 1993, [1993] Proceedings of the IEEE International Symposium on Requirements Engineering.

[16]  William E. Howden,et al.  Weak Mutation Testing and Completeness of Test Sets , 1982, IEEE Transactions on Software Engineering.

[17]  David Lorge Parnas,et al.  Software Requirements for the A-7E Aircraft. , 1992 .

[18]  Elaine J. Weyuker,et al.  Automatically Generating Test Data from a Boolean Specification , 1994, IEEE Trans. Software Eng..

[19]  Tatsuhiro Tsuchiya,et al.  On fault classes and error detection capability of specification-based testing , 2002, TSEM.

[20]  Man F. Lau,et al.  An extended fault class hierarchy for specification-based testing , 2005, TSEM.

[21]  Yuen-Tak Yu,et al.  On the relationships of faults for Boolean specification based testing , 2001, Proceedings 2001 Australian Software Engineering Conference.

[22]  Larry J Morell,et al.  A Theory of Fault-Based Testing , 1990, IEEE Trans. Software Eng..

[23]  Xin Feng,et al.  MIST: towards a minimum set of test cases , 2002 .

[24]  Elaine J. Weyuker,et al.  More Experience with Data Flow Testing , 1993, IEEE Trans. Software Eng..

[25]  Elaine J. Weyuker,et al.  Analyzing Partition Testing Strategies , 1991, IEEE Trans. Software Eng..

[26]  Kuo-Chung Tai,et al.  Theory of Fault-Based Predicate Testing for Computer Programs , 1996, IEEE Trans. Software Eng..

[27]  David Lorge Parnas,et al.  Functional Documents for Computer Systems , 1995, Sci. Comput. Program..

[28]  Paul Ammann,et al.  Reconciling perspectives of software logic testing , 2008 .

[29]  David Lorge Parnas,et al.  Tabular Representation of Relations , 1992 .

[30]  David Lorge Parnas,et al.  Precise Documentation of Well-Structured Programs , 1994, IEEE Trans. Software Eng..

[31]  Wynne Hsu,et al.  DESIGN OF MUTANT OPERATORS FOR THE C PROGRAMMING LANGUAGE , 2006 .

[32]  Elaine J. Weyuker,et al.  A Formal Analysis of the Fault-Detecting Ability of Testing Methods , 1993, IEEE Trans. Software Eng..

[33]  Paul C. Jorgensen,et al.  Software Testing: A Craftsman's Approach , 1995 .

[34]  Marc J. Balcer,et al.  The category-partition method for specifying and generating fuctional tests , 1988, CACM.

[35]  David Lorge Parnas,et al.  Tabular Representations in Relational Documents , 1997, Relational Methods in Computer Science.

[36]  Tsong Yueh Chen,et al.  On the statistical properties of testing effectiveness measures , 2006, J. Syst. Softw..

[37]  Tsong Yueh Chen,et al.  On the Expected Number of Failures Detected by Subdomain Testing and Random Testing , 1996, IEEE Trans. Software Eng..

[38]  Xin Feng,et al.  ESTP: An Experimental Software Testing Platform , 2008, Testing: Academic & Industrial Conference - Practice and Research Techniques (taic part 2008).

[39]  David Lorge Parnas,et al.  Assessment of safety-critical software in nuclear power plants , 1991 .

[40]  David Lorge Parnas,et al.  Using information about functions in selecting test cases , 2005, A-MOST '05.

[41]  Jeffery I. Zucker.  Transformations of normal and inverted function tables , 2005, Formal Aspects of Computing.

[42]  David Lorge Parnas,et al.  Specification of Software Component Requirements Using the Trace Function Method , 2006, 2006 International Conference on Software Engineering Advances (ICSEA'06).

[43]  Victor R. Basili,et al.  Comparing the Effectiveness of Software Testing Strategies , 1987, IEEE Transactions on Software Engineering.

[44]  Vadim Okun,et al.  Comparison of fault classes in specification-based testing , 2004, Inf. Softw. Technol..

[45]  Yuen-Tak Yu,et al.  A comparison of MC/DC, MUMCUT and several other coverage criteria for logical decisions , 2006, J. Syst. Softw..

[46]  Tsong Yueh Chen,et al.  MUMCUT: a fault-based strategy for testing Boolean specifications , 1999, Proceedings Sixth Asia Pacific Software Engineering Conference (ASPEC'99) (Cat. No.PR00509).

[47]  P. Halmos.  Naive Set Theory , 1961 .

[48]  Richard Lai,et al.  Analyzing and Extending MUMCUT for Fault-based Testing of General Boolean Expressions , 2006, The Sixth IEEE International Conference on Computer and Information Technology (CIT'06).

[49]  Tsong Yueh Chen,et al.  Test case selection strategies based on Boolean specifications ‡ , 2001, Softw. Test. Verification Reliab..