Test Suite Minimization of Evolving Software Systems: A Case Study

Test suite minimization ensures that an optimum set of test cases are selected to provide maximum coverage of requirements. In this paper, we discuss and evaluate techniques for test suite minimization of evolving software systems. As a case study, we have used an industrial tool, Static Code Analysis (SCAN) tool for Electronic Device Description Language (EDDL) as the System Under Test (SUT). We have used standard approaches including Greedy, Greedy Essential (GE) and Greedy Redundant Essential (GRE) for minimization of the test suite for a given set of requirements of the SUT. Further, we have proposed and implemented k-coverage variants of these approaches. The minimized test suite which is obtained as a result reduces testing effort and time during regression testing. The paper also addresses the need for choosing an appropriate level of granularity of requirements to efficiently cover all requirements. The paper demonstrates how fine grained requirements help in finding an optimal test suite to completely address the requirements and also help in detecting bugs in each version of the software. Finally, the results from different analyses have been presented and compared and it has been observed that GE heuristics performs the best (run time) under certain conditions.

[1]  Tsong Yueh Chen,et al.  Dividing Strategies for the Optimization of a Test Suite , 1996, Inf. Process. Lett..

[2]  Abhik Roychoudhury,et al.  CoREBench: studying complexity of regression errors , 2014, ISSTA 2014.

[3]  Emanuel Melachrinoudis,et al.  Bi-criteria models for all-uses test suite reduction , 2004, Proceedings. 26th International Conference on Software Engineering.

[4]  J. R. Horgan,et al.  A data flow coverage testing tool for C , 1992, [1992] Proceedings of the Second Symposium on Assessment of Quality Software Development Tools.

[5]  Michael D. Ernst,et al.  Improving test suites via operational abstraction , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[6]  Alessandro Orso,et al.  MINTS: A general framework and tool for supporting test-suite minimization , 2009, 2009 IEEE 31st International Conference on Software Engineering.

[7]  Sreeja Nair,et al.  A Generic Static Analysis Framework for Domain-specific Languages , 2018, 2018 IEEE 23rd International Conference on Emerging Technologies and Factory Automation (ETFA).

[8]  Kenneth Steiglitz,et al.  Combinatorial Optimization: Algorithms and Complexity , 1981 .

[9]  T. Y. Chen,et al.  Heuristics Towards The Optimization Of TheSize Of A Test Suite , 1970 .

[10]  Bruno C. d. S. Oliveira,et al.  Partition-based regression verification , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[11]  Bogdan Korel,et al.  Black-box test reduction using input-output analysis , 2000, ISSTA '00.

[12]  Neelam Gupta,et al.  A concept analysis inspired greedy algorithm for test suite minimization , 2005, PASTE '05.

[13]  George Mason,et al.  Procedures for Reducing the Size of Coverage-based Test Sets , 1995 .

[14]  Mark Harman,et al.  Pareto efficient multi-objective test case selection , 2007, ISSTA '07.

[15]  Neelam Gupta,et al.  Test suite reduction with selective redundancy , 2005, 21st IEEE International Conference on Software Maintenance (ICSM'05).

[16]  Abhik Roychoudhury,et al.  On Testing Embedded Software , 2016, Adv. Comput..

[17]  Mark Harman,et al.  Regression testing minimization, selection and prioritization: a survey , 2012, Softw. Test. Verification Reliab..

[18]  Atif M. Memon,et al.  Call-Stack Coverage for GUI Test Suite Reduction , 2008, IEEE Trans. Software Eng..

[19]  Antonia Bertolino,et al.  Using Spanning Sets for Coverage Testing , 2003, IEEE Trans. Software Eng..

[20]  Neelam Gupta,et al.  Improving Fault Detection Capability by Selectively Retaining Test Cases during Test Suite Reduction , 2007, IEEE Transactions on Software Engineering.