When can we test less?

When it is impractical to rigorously assess all parts of complex systems, test engineers use defect detectors to focus their limited resources. We define some properties of an ideal defect detector and assess different methods of generating one. In the case study presented here, traditional methods of generating such detectors (e.g. reusing detectors from the literature, linear regression, model trees) were found to be inferior to those found via a PACE analysis.

[1]  Maurice H. Halstead,et al.  Elements of software science , 1977 .

[2]  Tim Menzies,et al.  Fast formal analysis of requirements via "topoi diagrams" , 2001, Proceedings of the 23rd International Conference on Software Engineering. ICSE 2001.

[3]  Steven R. Rakitin,et al.  Software verification and validation for practitioners and managers , 2001 .

[4]  K. Adlassnig,et al.  Performance evaluation of medical expert systems using ROC curves. , 1989, Computers and biomedical research, an international journal.

[5]  J. R. Quinlan.  Learning With Continuous Classes , 1992 .

[6]  Ron Kohavi,et al.  The Case against Accuracy Estimation for Comparing Induction Algorithms , 1998, ICML.

[7]  Michael R. Lowry,et al.  Towards a theory for integration of mathematical verification and empirical testing , 1998, Proceedings 13th IEEE International Conference on Automated Software Engineering (Cat. No.98EX239).

[8]  Norman E. Fenton,et al.  Software Metrics: A Rigorous Approach , 1991 .

[9]  Ian H. Witten,et al.  Data mining: practical machine learning tools and techniques with Java implementations , 2002, SGMD.

[10]  Anas N. Al-Rabadi,et al.  A comparison of modified reconstructability analysis and Ashenhurst‐Curtis decomposition of Boolean functions , 2004 .

[11]  Leo Breiman,et al.  Classification and Regression Trees , 1984 .

[12]  T. Menzies,et al.  Metrics that matter , 2002, 27th Annual NASA Goddard/IEEE Software Engineering Workshop, 2002. Proceedings..

[13]  Bojan Cukic,et al.  How Many Tests are Enough , 2000 .