How good is your blind spot sampling policy

Assessing software costs money and better assessment costs exponentially more money. Given finite budgets, assessment resources are typically skewed towards areas that are believed to be mission critical. This leaves blind spots: portions of the system that may contain defects which may be missed. Therefore, in addition to rigorously assessing mission critical areas, a parallel activity should sample the blind spots. This paper assesses defect detectors based on static code measures as a blind spot sampling method. In contrast to previous results, we find that such defect detectors yield results that are stable across many applications. Further, these detectors are inexpensive to use and can be tuned to the specifics of the current business situations.

[1]  Norman E. Fenton,et al.  Software Metrics: A Rigorous Approach , 1991 .

[2]  Aiko M. Hormann,et al.  Programs for Machine Learning. Part I , 1962, Inf. Control..

[3]  Tim Menzies,et al.  When can we test less? , 2003, Proceedings. 5th International Workshop on Enterprise Networking and Computing in Healthcare Industry (IEEE Cat. No.03EX717).

[4]  Ian Witten,et al.  Data Mining , 2000 .

[5]  Michael R. Lowry,et al.  Towards a theory for integration of mathematical verification and empirical testing , 1998, Proceedings 13th IEEE International Conference on Automated Software Engineering (Cat. No.98EX239).

[6]  Ron Kohavi,et al.  The Case against Accuracy Estimation for Comparing Induction Algorithms , 1998, ICML.

[7]  A. P. Nikora,et al.  How simple is software defect detection , 2003 .

[8]  Bojan Cukic,et al.  How Many Tests are Enough , 2000 .

[9]  J. Voas,et al.  Software Testability: The New Verification , 1995, IEEE Softw..

[10]  Tim Menzies,et al.  Fast formal analysis of requirements via "topoi diagrams" , 2001, Proceedings of the 23rd International Conference on Software Engineering. ICSE 2001.

[11]  Darrel C. Ince,et al.  A critique of three metrics , 1994, J. Syst. Softw..

[12]  K. Adlassnig,et al.  Performance evaluation of medical expert systems using ROC curves. , 1989, Computers and biomedical research, an international journal.

[13]  Shari Lawrence Pfleeger,et al.  Software metrics (2nd ed.): a rigorous and practical approach , 1997 .

[14]  J. R. Quinlan.  Learning With Continuous Classes , 1992 .

[15]  Robyn R. Lutz,et al.  Operational anomalies as a cause of safety-critical requirements evolution , 2003, J. Syst. Softw..

[16]  J. Ross Quinlan,et al.  C4.5: Programs for Machine Learning , 1992 .

[17]  Ian H. Witten,et al.  Data mining: practical machine learning tools and techniques with Java implementations , 2002, SGMD.

[18]  Nancy G. Leveson,et al.  Safeware: System Safety and Computers , 1995 .