Detecting and Reducing Redundancy in Software Testing for Highly Configurable Systems

While redundancy in software development has beenused as a fault tolerance mechanism that can be useful forincreasing software quality, redundancy in test artifacts leads to decreased testing performance and increased effort, mainly due to repetitive testing and high test maintenance costs. The effect of redundancy is especially accentuated in testing software for highly configurable systems, which contain a large number of configurable mutually interacting options that overlap across a test suite and introduce unnecessary redundancy. In this paper we propose a methodology for detecting and reducing redundancy in test suits for highly configurable software. We use equivalence partitioning to segregate test cases covering the same sets of feature interactions, then identify partitions with multiple test cases, and finally we remove those test cases that do not have unique partitions. We evaluate the approach in the industrial case study of testing highly configurable software, and on a large set of test suites developed motivated by the industrial case study. The results show that our approach can reduce testexecution time by 35% on average compared to industry practice, without significant effect on fault detection effectiveness.

[1]  Vahid Garousi,et al.  A Tester-Assisted Methodology for Test Redundancy Detection , 2010, Adv. Softw. Eng..

[2]  Gregg Rothermel,et al.  An empirical study of the effects of minimization on the fault detection capabilities of test suites , 1998, Proceedings. International Conference on Software Maintenance (Cat. No. 98CB36272).

[3]  Jaejoon Lee,et al.  FORM: A feature-;oriented reuse method with domain-;specific reference architectures , 1998, Ann. Softw. Eng..

[4]  Gordon Fraser,et al.  Redundancy Based Test-Suite Reduction , 2007, FASE.

[5]  David Leon,et al.  An Empirical Study of Test Case Filtering Techniques Based on Exercising Information Flows , 2007, IEEE Transactions on Software Engineering.

[6]  Neelam Gupta,et al.  Improving Fault Detection Capability by Selectively Retaining Test Cases during Test Suite Reduction , 2007, IEEE Transactions on Software Engineering.

[7]  Mats Per Erik Heimdahl,et al.  Test-suite reduction for model based tests: effects on test quality and implications for testing , 2004, Proceedings. 19th International Conference on Automated Software Engineering, 2004..

[8]  Arnaud Gotlieb,et al.  Test Case Prioritization for Continuous Regression Testing: An Industrial Case Study , 2013, 2013 IEEE International Conference on Software Maintenance.

[9]  Vahid Garousi,et al.  Test Redundancy Measurement Based on Coverage Information: Evaluations and Lessons Learned , 2009, 2009 International Conference on Software Testing Verification and Validation.

[10]  Gregg Rothermel,et al.  Empirical studies of test‐suite reduction , 2002, Softw. Test. Verification Reliab..

[11]  George Mason,et al.  Procedures for Reducing the Size of Coverage-based Test Sets , 1995 .