Beyond code coverage — An approach for test suite assessment and improvement

Code coverage is successfully used to guide white box test design and evaluate the respective test completeness. However, simple overall coverage ratios are often not precise enough to effectively help when a (regression) test suite needs to be reassessed and evolved after software change. We present an approach for test suite assessment and improvement that utilizes code coverage information, but on a more detailed level and adds further evaluation aspects derived from the coverage. The main use of the method is to aid various test suite evolution situations such as removal, refactoring and extension of test cases as a result of code change or test suite efficiency enhancement. We define various metrics to express different properties of test suites beyond simple code coverage ratios, and present the assessment and improvement process as an iterative application of different improvement goals and more specific sub-activities. The method is demonstrated by applying it to improve the tests of one of our experimental systems.

[1]  Joost Visser,et al.  A Practical Model for Measuring Maintainability , 2007, 6th International Conference on the Quality of Information and Communications Technology (QUATIC 2007).

[2]  Tibor Gyimóthy,et al.  Test suite reduction for fault detection and localization: A combined approach , 2014, 2014 Software Evolution Week - IEEE Conference on Software Maintenance, Reengineering, and Reverse Engineering (CSMR-WCRE).

[3]  Arie van Deursen,et al.  Refactoring test code , 2001 .

[4]  A.J.C. van Gemund,et al.  On the Accuracy of Spectrum-based Fault Localization , 2007, Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007).

[5]  Andy Zaidman,et al.  Test Code Quality and Its Relation to Issue Handling Performance , 2014, IEEE Transactions on Software Engineering.

[6]  B. Van Rompaey,et al.  On The Detection of Test Smells: A Metrics-Based Approach for General Fixture and Eager Test , 2007, IEEE Transactions on Software Engineering.

[7]  Rini van Solingen,et al.  Goal Question Metric (GQM) Approach , 2002 .

[8]  Hong Zhu,et al.  Software unit test coverage and adequacy , 1997, ACM Comput. Surv..

[9]  Akbar Siami Namin,et al.  The influence of size and coverage on test suite effectiveness , 2009, ISSTA.

[10]  Mark Harman,et al.  Regression testing minimization, selection and prioritization: a survey , 2012, Softw. Test. Verification Reliab..

[11]  Mark Harman,et al.  Coverage and fault detection of the output-uniqueness test selection criteria , 2014, ISSTA 2014.

[12]  Tibor Gyimóthy,et al.  Toolset and Program Repository for Code Coverage-Based Test Suite Analysis and Manipulation , 2014, 2014 IEEE 14th International Working Conference on Source Code Analysis and Manipulation.

[13]  Angélica Caro,et al.  A Probabilistic Approach to Web Portal's Data Quality Evaluation , 2007 .

[14]  Yuri Chernak,et al.  Validating and Improving Test-Case Effectiveness , 2001, IEEE Softw..

[15]  Antonia Bertolino,et al.  Software Testing Research: Achievements, Challenges, Dreams , 2007, Future of Software Engineering (FOSE '07).

[16]  Rui Abreu,et al.  A diagnosis-based approach to software comprehension , 2014, ICPC 2014.

[17]  Alessandro Orso,et al.  Understanding myths and realities of test-suite evolution , 2012, SIGSOFT FSE.

[18]  Pekka Abrahamsson,et al.  Providing test quality feedback using static source code and automatic test suite metrics , 2005, 16th IEEE International Symposium on Software Reliability Engineering (ISSRE'05).

[19]  Doreen Meier,et al.  Agile Testing A Practical Guide For Testers And Agile Teams , 2016 .

[20]  Lee Copeland,et al.  A Practitioner's Guide to Software Test Design , 2003 .

[21]  Mario Piattini,et al.  A systematic review measurement in software engineering: State-of-the-art in Measures , 2006, ICSOFT.

[22]  Reid Holmes,et al.  Coverage is not strongly correlated with test suite effectiveness , 2014, ICSE.

[23]  Gabriele Bavota,et al.  An empirical analysis of the distribution of unit test smells and their impact on software maintenance , 2012, 2012 28th IEEE International Conference on Software Maintenance (ICSM).