Automated Search for Good Coverage Criteria: Moving from Code Coverage to Fault Coverage through Search-Based Software Engineering

We propose to use Search-Based Software Engineering to automatically evolve coverage criteria that are well correlated with fault revelation, through the use of existing fault databases. We explain how problems of bloat and overfitting can be ameliorated in our approach, and show how this new method will yield insight into faults --- as well as better guidance for Search-Based Software Testing.

[1]  L. Altenberg.  EMERGENT PHENOMENA IN GENETIC PROGRAMMING , 1994 .

[2]  Luciano Baresi,et al.  An Introduction to Software Testing , 2006, FoVMT.

[3]  Tatsuhiro Tsuchiya,et al.  On fault classes and error detection capability of specification-based testing , 2002, TSEM.

[4]  Luciano Lavagno,et al.  Electronic Design Automation for Integrated Circuits Handbook , 2006 .

[5]  Gordon Fraser,et al.  Does Automated Unit Test Generation Really Help Software Testers? A Controlled Empirical Study , 2015, ACM Trans. Softw. Eng. Methodol..

[6]  Andreas Zeller,et al.  Simplifying and Isolating Failure-Inducing Input , 2002, IEEE Trans. Software Eng..

[7]  Michael D. Ernst,et al.  Defects4J: a database of existing faults to enable controlled testing studies for Java programs , 2014, ISSTA 2014.

[8]  Phil McMinn,et al.  The Effectiveness of Test Coverage Criteria for Relational Database Schema Integrity Constraints , 2015, ACM Trans. Softw. Eng. Methodol..

[9]  Mark Harman,et al.  Search-based amorphous slicing , 2005, 12th Working Conference on Reverse Engineering (WCRE'05).

[10]  Reid Holmes,et al.  Coverage is not strongly correlated with test suite effectiveness , 2014, ICSE.

[11]  Alessandro Orso,et al.  Search-Based Propagation of Regression Faults in Automated Regression Testing , 2013, 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation Workshops.

[12]  Inderpal S. Bhandari,et al.  Orthogonal Defect Classification - A Concept for In-Process Measurements , 1992, IEEE Trans. Software Eng..

[13]  Phil McMinn,et al.  Search‐based software test data generation: a survey , 2004, Softw. Test. Verification Reliab..

[14]  Gordon Fraser,et al.  Do Automatically Generated Unit Tests Find Real Faults? An Empirical Study of Effectiveness and Challenges (T) , 2015, 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[15]  Gregory Gay,et al.  The Risks of Coverage-Directed Test Case Generation , 2015, IEEE Transactions on Software Engineering.

[16]  Mark Harman,et al.  Ieee Transactions on Evolutionary Computation 1 , 2022 .

[17]  Jane Huffman Hayes.  Building a requirement fault taxonomy: experiences from a NASA verification and validation research project , 2003, 14th International Symposium on Software Reliability Engineering, 2003. ISSRE 2003..

[18]  Abhik Roychoudhury,et al.  CoREBench: studying complexity of regression errors , 2014, ISSTA 2014.

[19]  A. Jefferson Offutt,et al.  Introduction to Software Testing , 2008 .

[20]  Mark Harman,et al.  The Current State and Future of Search Based Software Engineering , 2007, Future of Software Engineering (FOSE '07).

[21]  Gordon Fraser,et al.  Does automated white-box test generation really help software testers? , 2013, ISSTA.