Comparing the Effectiveness of Testing Techniques

Testing software systems requires practitioners to decide how to select test data. This chapter discusses what it means for one test data selection criterion to be more effective than another. Several proposed comparison relations are discussed, highlighting the strengths and weaknesses of each. Also included is a discussion of how these relations evolved and argue that large scale empirical studies are needed.

[1]  Elaine J. Weyuker,et al.  Axiomatizing software test data adequacy , 1986, IEEE Transactions on Software Engineering.

[2]  Elaine J. Weyuker,et al.  Theories of Program Testing and the Application of Revealing Subdomains , 1980, IEEE Transactions on Software Engineering.

[3]  Yves Crouzet,et al.  An experimental study on software structural testing: deterministic versus random input generation , 1991, [1991] Digest of Papers. Fault-Tolerant Computing: The Twenty-First International Symposium.

[4]  Elaine J. Weyuker,et al.  Testing software to detect and reduce risk , 2000, J. Syst. Softw..

[5]  Richard G. Hamlet,et al.  Theoretical comparison of testing methods , 1989, TAV3.

[6]  R. Taylor,et al.  Partition testing does not inspire confidence , 1988, [1988] Proceedings. Second Workshop on Software Testing, Verification, and Analysis.

[7]  Elaine J. Weyuker,et al.  Deriving Workloads for Performance Testing , 1996, Softw. Pract. Exp..

[8]  Robert M. Hierons.  Comparing test sets and criteria in the presence of test hypotheses and fault domains , 2002, TSEM.

[9]  A. Jefferson Offutt,et al.  An evaluation of combination strategies for test case selection , 2006, Empirical Software Engineering.

[10]  Lorenzo Strigini,et al.  Comparing the effectiveness of testing methods in improving programs: the effect of variations in program quality , 1998, Proceedings Ninth International Symposium on Software Reliability Engineering (Cat. No.98TB100257).

[11]  Elaine J. Weyuker,et al.  Evaluating Software Complexity Measures , 2010, IEEE Trans. Software Eng..

[12]  Elaine J. Weyuker,et al.  Data flow analysis techniques for test data selection , 2015, ICSE '82.

[13]  Elaine J. Weyuker,et al.  A Formal Analysis of the Fault-Detecting Ability of Testing Methods , 1993, IEEE Trans. Software Eng..

[14]  A. Jefferson Offutt,et al.  An Experimental Evaluation of Data Flow and Mutation Testing , 1996 .

[15]  Elaine J. Weyuker,et al.  Analyzing Partition Testing Strategies , 1991, IEEE Trans. Software Eng..

[16]  Simeon C. Ntafos,et al.  A Comparison of Some Structural Testing Strategies , 1988, IEEE Trans. Software Eng..

[17]  Phyllis G. Frankl,et al.  All-uses vs mutation testing: An experimental comparison of effectiveness , 1997, J. Syst. Softw..

[18]  Phyllis G. Frankl,et al.  An Experimental Comparison of the Effectiveness of Branch Testing and Data Flow Testing , 1993, IEEE Trans. Software Eng..

[19]  S. N. Weiss,et al.  All-Uses versus Mutation Testing : An ExperimentalComparison of E ectiveness , 1996 .

[20]  Bev Littlewood,et al.  Evaluating Testing Methods by Delivered Reliability , 1998, IEEE Trans. Software Eng..

[21]  M. J. Reilly,et al.  An investigation of the applicability of design of experiments to software testing , 2002, 27th Annual NASA Goddard/IEEE Software Engineering Workshop, 2002. Proceedings..

[22]  Elaine J. Weyuker,et al.  Predicting the location and number of faults in large software systems , 2005, IEEE Transactions on Software Engineering.

[23]  Elaine J. Weyuker,et al.  Selecting Software Test Data Using Data Flow Information , 1985, IEEE Transactions on Software Engineering.

[24]  Thomas J. Ostrand,et al.  Experiments on the effectiveness of dataflow- and control-flow-based test adequacy criteria , 1994, Proceedings of 16th International Conference on Software Engineering.

[25]  Elaine J. Weyuker,et al.  A metric to predict software scalability , 2002, Proceedings Eighth IEEE Symposium on Software Metrics.

[26]  Elaine J. Weyuker,et al.  Provable Improvements on Branch Testing , 1993, IEEE Trans. Software Eng..

[27]  John S. Gourlay.  A Mathematical Framework for the Investigation of Testing , 1983, IEEE Transactions on Software Engineering.

[28]  Elaine J. Weyuker,et al.  Comparison of program testing strategies , 1991, TAV4.

[29]  Elaine J. Weyuker.  Using failure cost information for testing and reliability assessment , 1996, TSEM.

[30]  Elaine J. Weyuker,et al.  Metrics to Assess the Likelihood of Project Success Based on Architecture Reviews , 2004, Empirical Software Engineering.

[31]  Simeon C. Ntafos,et al.  An Evaluation of Random Testing , 1984, IEEE Transactions on Software Engineering.

[32]  Elaine J. Weyuker,et al.  The Automatic Generation of Load Test Suites and the Assessment of the Resulting Software , 1995, IEEE Trans. Software Eng..