Context-specific Quality Evaluation of Test Cases

Software systems are continuously changed during maintenance and evolution. To ensure their quality, they have to be tested. But before starting testing a software system, the quality of the test cases themselves has to be evaluated. Due to the changes of the software system, they might have become obsolete or even erroneous. Furthermore, test cases created in an industrial setting are extensive and at some point of time, they might have become difficult to understand, unmanageable and inefficient. Therefore, by evaluating their quality, we can better understand, control and eventually improve the quality of test cases. We present the Test Case Quality Plan (TCQP) approach, which is based on the GQM (Goal-Question-Metric) approach and enables a systematic and efficient development of quality plans. They serve as a guideline for the quality evaluation of test cases, and emphasize the context of use of test cases as a major factor of influence for the whole quality evaluation. The TCQP approach has been applied and evaluated in an industrial case study.

[1]  Lionel C. Briand,et al.  Practical guidelines for measurement-based process improvement , 1996, Softw. Process. Improv. Pract..

[2]  Audris Mockus,et al.  Test coverage and post-verification defects: A multiple case study , 2009, ESEM 2009.

[3]  Mats Per Erik Heimdahl,et al.  Programs, tests, and oracles: the foundations of testing revisited , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[4]  Gerard Meszaros,et al.  xUnit Test Patterns: Refactoring Test Code , 2007 .

[5]  Ina Schieferdecker,et al.  Applying the ISO 9126 quality model to test specifications - exemplified for TTCN-3 test specifications , 2007, Software Engineering.

[6]  Burak Turhan,et al.  How Good Are My Tests? , 2017, 2017 IEEE/ACM 8th Workshop on Emerging Trends in Software Metrics (WETSoM).

[7]  Eduardo Martins Guerra,et al.  Refactoring Test Code Safely , 2007, International Conference on Software Engineering Advances (ICSEA 2007).

[8]  Arie van Deursen,et al.  Refactoring test code , 2001 .

[9]  Maximilian Junker,et al.  Hunting for smells in natural language tests , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[10]  Stefan Wagner,et al.  Multi-Dimensional Measures for Test Case Quality , 2008, 2008 IEEE International Conference on Software Testing Verification and Validation Workshop.

[11]  Cem Kaner,et al.  What IS a Good Test Case , 2003 .

[12]  Harry M. Sneed.  Measuring the Effectiveness of Software Testing , 2004, SOQUA/TECOS.

[13]  Gregor Engels,et al.  Kontextsensitive Qualitätsplanung für Software-Modelle , 2008, Modellierung.

[14]  Alex Groce,et al.  Code coverage for suite evaluation by developers , 2014, ICSE.

[15]  Michael Ellims,et al.  The Economics of Unit Testing , 2006, Empirical Software Engineering.

[16]  Yuri Chernak,et al.  Validating and Improving Test-Case Effectiveness , 2001, IEEE Softw..

[17]  Gregor Engels,et al.  Quality Plans for Measuring Testability of Models , 2008 .

[18]  ISO / IEC 25010 : 2011 Systems and software engineering — Systems and software Quality Requirements and Evaluation ( SQuaRE ) — System and software quality models , 2013 .

[19]  Mauricio A. Saca.  Refactoring improving the design of existing code , 2017, 2017 IEEE 37th Central America and Panama Convention (CONCAPAN XXXVII).