Test case prioritization and selection technique in continuous integration development environments: a case study

Regression testing is a very important activity in continuous integration development environments. Software engineers frequently integrate new or changed code that involves in a new regression testing. Furthermore, regression testing in continuous integration development environments is together with tight time constraints. It is also impossible to re-run all the test cases in regression testing. Test case prioritization and selection technique are often used to render continuous integration processes more cost-effective. According to multi objective optimization, we present a test case prioritization and selection technique, TCPSCI, to satisfy time constraints and achieve testing goals in continuous integration development environments. Based on historical failure data, testing coverage code size and testing execution time, we order and select test cases. The test cases of the maximize code coverage, the shorter execution time and revealing the latest faults have the higher priority in the same change request. The case study results show that using TCPSCI has a higher costeffectiveness comparing to the manually prioritization.

[1]  Gregg Rothermel,et al.  Analyzing Regression Test Selection Techniques , 1996, IEEE Trans. Software Eng..

[2]  Mark Harman,et al.  Regression testing minimization, selection and prioritization: a survey , 2012, Softw. Test. Verification Reliab..

[3]  Anne M. Denton,et al.  A clustering approach to improving test case prioritization: An industrial case study , 2011, 2011 27th IEEE International Conference on Software Maintenance (ICSM).

[4]  Morten Mossige,et al.  Reinforcement learning for automatic test case prioritization and selection in continuous integration , 2017, ISSTA.

[5]  Andrew Glover,et al.  Continuous Integration: Improving Software Quality and Reducing Risk (The Addison-Wesley Signature Series) , 2007 .

[6]  Gregg Rothermel,et al.  Prioritizing test cases for regression testing , 2000, ISSTA '00.

[7]  Bo Qu,et al.  Test Case Prioritization Based on Test Suite Design Information: Test Case Prioritization Based on Test Suite Design Information , 2009 .

[8]  Hyunsook Do,et al.  Test Case Prioritization Using Requirements-Based Clustering , 2013, 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation.

[9]  Arnaud Gotlieb,et al.  Test Case Prioritization for Continuous Regression Testing: An Industrial Case Study , 2013, 2013 IEEE International Conference on Software Maintenance.

[10]  Gregg Rothermel,et al.  A safe, efficient regression test selection technique , 1997, TSEM.

[11]  Yves Le Traon,et al.  Comparing White-Box and Black-Box Test Prioritization , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE).

[12]  Gregg Rothermel,et al.  Test case prioritization: an empirical study , 1999, Proceedings IEEE International Conference on Software Maintenance - 1999 (ICSM'99). 'Software Maintenance for Business Change' (Cat. No.99CB36360).

[13]  P. V. R. D. Prasad,et al.  Survey on Test Case Prioritization Techniques for Regression Testing , 2017 .

[14]  Mark Harman,et al.  Faster Fault Finding at Google Using Multi Objective Regression Test Optimisation , 2011 .

[15]  Sherin M. Moussa,et al.  Cluster‐based test cases prioritization and selection technique for agile regression testing , 2017, J. Softw. Evol. Process..

[16]  Gregg Rothermel,et al.  Techniques for improving regression testing in continuous integration development environments , 2014, SIGSOFT FSE.

[17]  Gregg Rothermel,et al.  Modeling the cost-benefits tradeoffs for regression testing techniques , 2002, International Conference on Software Maintenance, 2002. Proceedings..