A Controlled Experiment to Evaluate Effectiveness and Efficiency of Three Software Testing Methods

In this paper, we describe a controlled experiment carried out to compare three software testing methods: code reading, functional testing and structural testing. The experiment was carried out with eighteen subjects who applied three techniques to three C programs in a fractional factorial experimental design. The main results of the study are that all testing techniques are equivalent in terms of effectiveness; however the techniques differ partially in terms of efficiency.

[1]  Jeffrey C. Carver.  Towards Reporting Guidelines for Experimental Replications: A Proposal , 2010 .

[2]  Erik Kamsties,et al.  An Empirical Evaluation of Three Defect-Detection Techniques , 1995, ESEC.

[3]  R BasiliVictor,et al.  Comparing the Effectiveness of Software Testing Strategies , 1987 .

[4]  Christopher M. Lott,et al.  Repeatable software engineering experiments for comparing defect-detection techniques , 2004, Empirical Software Engineering.

[5]  Victor R. Basili,et al.  Experimentation in software engineering , 1986, IEEE Transactions on Software Engineering.

[6]  Natalia Juristo Juzgado,et al.  A Look at 25 Years of Data , 2009, IEEE Software.

[7]  Victor R. Basili,et al.  Comparing the Effectiveness of Software Testing Strategies , 1987, IEEE Transactions on Software Engineering.

[8]  Natalia Juristo Juzgado,et al.  Reviewing 25 Years of Testing Technique Experiments , 2004, Empirical Software Engineering.

[9]  Victor R. Basili,et al.  Experimental Software Engineering Issues: Critical Assessment and Future Directions , 1993, Lecture Notes in Computer Science.

[10]  James Miller,et al.  An empirical evaluation of defect detection techniques , 1997, Inf. Softw. Technol..

[11]  Natalia Juristo Juzgado,et al.  Functional Testing, Structural Testing, and Code Reading: What Fault Type Do They Each Detect? , 2003, ESERNET.

[12]  Victor R. Basili.  The Role of Controlled Experiments in Software Engineering Research , 2006, Empirical Software Engineering Issues.