REMAP: Using Rule Mining and Multi-objective Search for Dynamic Test Case Prioritization

Test case prioritization (TP) prioritizes test cases into an optimal order for achieving specific criteria (e.g., higher fault detection capability) as early as possible. However, the existing TP techniques usually only produce a static test case order before the execution without taking runtime test case execution results into account. In this paper, we propose an approach for black-box dynamic TP using rule mining and multi-objective search (named as REMAP). REMAP has three key components: 1) Rule Miner, which mines execution relations among test cases from historical execution data; 2) Static Prioritizer, which defines two objectives (i.e., fault detection capability (FDC) and test case reliance score (TRS)) and applies multi-objective search to prioritize test cases statically; and 3) Dynamic Executor and Prioritizer, which executes statically-prioritized test cases and dynamically updates the test case order based on the runtime test case execution results. We empirically evaluated REMAP with random search, greedy based on FDC, greedy based on FDC and TRS, static search-based prioritization, and rule-based prioritization using two industrial and three open source case studies. Results showed that REMAP significantly outperformed the other approaches for 96% of the case studies and managed to achieve on average 18% higher Average Percentage of Faults Detected (APFD).

[1]  Gordon Fraser,et al.  On Parameter Tuning in Search Based Software Engineering , 2011, SSBSE.

[2]  H. B. Mann,et al.  On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other , 1947 .

[3]  Shuai Wang,et al.  Enhancing Test Case Prioritization in an Industrial Setting with Resource Awareness and Multi-objective Search , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering Companion (ICSE-C).

[4]  Sergio A. Alvarez,et al.  Efficient Adaptive-Support Association Rule Mining for Recommender Systems , 2004, Data Mining and Knowledge Discovery.

[5]  อนิรุธ สืบสิงห์,et al.  Data Mining Practical Machine Learning Tools and Techniques , 2014 .

[6]  Mark Harman,et al.  Search Algorithms for Regression Test Case Prioritization , 2007, IEEE Transactions on Software Engineering.

[7]  Tao Xie,et al.  Time-aware test-case prioritization using integer linear programming , 2009, ISSTA.

[8]  Durga Prasad Mohapatra,et al.  Test Case Prioritization Using Association Rule Mining and Business Criticality Test Value , 2016 .

[9]  Roberto J. Bayardo.  Brute-Force Mining of High-Confidence Classification Rules , 1997, KDD.

[10]  Saeed Parsa,et al.  Incorporating Historical Test Case Performance Data and Resource Constraints into Test Case Prioritization , 2009, TAP@TOOLS.

[11]  Sebastián Ventura,et al.  An interpretable classification rule mining algorithm , 2013, Inf. Sci..

[12]  Ladan Tahvildari,et al.  Size-Constrained Regression Test Case Selection Using Multicriteria Optimization , 2012, IEEE Transactions on Software Engineering.

[13]  Mohammed J. Zaki,et al.  Lazy Associative Classification , 2006, Sixth International Conference on Data Mining (ICDM'06).

[14]  Kichun Lee,et al.  Predictability-based collective class association rule mining , 2017, Expert Syst. Appl..

[15]  William W. Cohen.  Fast Effective Rule Induction , 1995, ICML.

[16]  Hadi Hemmati,et al.  A similarity-based approach for test case prioritization using historical failure data , 2015, 2015 IEEE 26th International Symposium on Software Reliability Engineering (ISSRE).

[17]  Neelam Gupta,et al.  A concept analysis inspired greedy algorithm for test suite minimization , 2005, PASTE '05.

[18]  Mary Jean Harrold,et al.  Recomputing Coverage Information to Assist Regression Testing , 2009, IEEE Transactions on Software Engineering.

[19]  Kalyanmoy Deb,et al.  A fast and elitist multiobjective genetic algorithm: NSGA-II , 2002, IEEE Trans. Evol. Comput..

[20]  Gregg Rothermel,et al.  Techniques for improving regression testing in continuous integration development environments , 2014, SIGSOFT FSE.

[21]  Jan Vanthienen,et al.  Software Defect Prediction Based on Association Rule Classification , 2010 .

[22]  Jun Cheng,et al.  A Fine-Grained Parallel Multi-objective Test Case Prioritization on GPU , 2013, SSBSE.

[23]  Bo Qu,et al.  Test Case Prioritization for Black Box Testing , 2007, 31st Annual International Computer Software and Applications Conference (COMPSAC 2007).

[24]  Gregg Rothermel,et al.  Test case prioritization: an empirical study , 1999, Proceedings IEEE International Conference on Software Maintenance - 1999 (ICSM'99). 'Software Maintenance for Business Change' (Cat. No.99CB36360).

[25]  Wei-Tek Tsai,et al.  Regression testing in an industrial environment , 1998, CACM.

[26]  Lothar Thiele,et al.  Comparison of Multiobjective Evolutionary Algorithms: Empirical Results , 2000, Evolutionary Computation.

[27]  Joseph Robert Horgan,et al.  A study of effective regression testing in practice , 1997, Proceedings The Eighth International Symposium on Software Reliability Engineering.

[28]  Shuai Wang,et al.  CBGA-ES: A Cluster-Based Genetic Algorithm with Elitist Selection for Supporting Multi-Objective Test Optimization , 2017, 2017 IEEE International Conference on Software Testing, Verification and Validation (ICST).

[29]  A. Vargha,et al.  A Critique and Improvement of the CL Common Language Effect Size Statistics of McGraw and Wong , 2000 .

[30]  Lionel C. Briand,et al.  A practical guide for using statistical tests to assess randomized algorithms in software engineering , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[31]  Antonio Ruiz Cortés,et al.  Multi-objective test case prioritization in highly configurable systems: A case study , 2016, J. Syst. Softw..

[32]  Shuai Wang,et al.  Search-Based Cost-Effective Test Case Selection within a Time Budget: An Empirical Study , 2016, GECCO.

[33]  Morten Mossige,et al.  Reinforcement learning for automatic test case prioritization and selection in continuous integration , 2017, ISSTA.

[34]  Kalyanmoy Deb,et al.  MULTI-OBJECTIVE FUNCTION OPTIMIZATION USING NON-DOMINATED SORTING GENETIC ALGORITHMS , 1994 .

[35]  Sunil Prabhakar,et al.  A Rule-Based Classification Algorithm for Uncertain Data , 2009, 2009 IEEE 25th International Conference on Data Engineering.

[36]  Shuai Wang,et al.  STIPI: Using Search to Prioritize Test Cases Based on Multi-objectives Derived from Industrial Practice , 2016, ICTSS.

[37]  Deepti Mishra,et al.  Test case prioritization: a systematic mapping study , 2012, Software Quality Journal.

[38]  Arnaud Gotlieb,et al.  Test Case Prioritization for Continuous Regression Testing: An Industrial Case Study , 2013, 2013 IEEE International Conference on Software Maintenance.

[39]  Lu Zhang,et al.  How Does Regression Test Prioritization Perform in Real-World Software Evolution? , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE).

[40]  Gregg Rothermel,et al.  A safe, efficient regression test selection technique , 1997, TSEM.

[41]  Yves Le Traon,et al.  Comparing White-Box and Black-Box Test Prioritization , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE).

[42]  Hyuncheol Park,et al.  Historical Value-Based Approach for Cost-Cognizant Test Case Prioritization to Improve the Effectiveness of Regression Testing , 2008, 2008 Second International Conference on Secure System Integration and Reliability Improvement.

[43]  Hyunsook Do,et al.  Improving the effectiveness of test suite through mining historical data , 2014, MSR 2014.

[44]  Gregg Rothermel,et al.  Incorporating varying test costs and fault severities into test case prioritization , 2001, Proceedings of the 23rd International Conference on Software Engineering. ICSE 2001.

[45]  Mark Harman,et al.  Multi-objective Software Effort Estimation , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE).

[46]  A. Dias-Neto,et al.  0006/2011 - Threats to Validity in Search-based Software Engineering Empirical Studies , 2011 .

[47]  Mary Lou Soffa,et al.  TimeAware test suite prioritization , 2006, ISSTA '06.

[48]  Mark Harman,et al.  Search Based Approaches to Component Selection and Prioritization for the Next Release Problem , 2006, 2006 22nd IEEE International Conference on Software Maintenance.

[49]  Durga Prasad Mohapatra,et al.  Model Based Test Case Prioritization Using Association Rule Mining , 2015 .

[50]  Tim Menzies,et al.  On the value of user preferences in search-based software engineering: A case study in software product lines , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[51]  Antonio J. Nebro,et al.  jMetal: A Java framework for multi-objective optimization , 2011, Adv. Eng. Softw..

[52]  Seyed Taghi Akhavan Niaki,et al.  A hybrid vendor managed inventory and redundancy allocation optimization problem in supply chain management: An NSGA-II with tuned parameters , 2014, Comput. Oper. Res..

[53]  Arnaud Gotlieb,et al.  Multi-objective test prioritization in software product line testing: an industrial case study , 2014, SPLC.

[54]  S. Vijayarani,et al.  An Efficient Algorithm for Generating Classification Rules , 2011 .

[55]  Mark Harman,et al.  Causal impact analysis for app releases in google play , 2016, SIGSOFT FSE.

[56]  Mark Harman,et al.  Regression testing minimization, selection and prioritization: a survey , 2012, Softw. Test. Verification Reliab..

[57]  Abdel Salam Sayyad,et al.  Pareto-optimal search-based software engineering (POSBSE): A literature survey , 2013, 2013 2nd International Workshop on Realizing Artificial Intelligence Synergies in Software Engineering (RAISE).

[58]  Mohammad Abdollahi Azgomi,et al.  An improved method for test case prioritization by incorporating historical test case data , 2012, Sci. Comput. Program..

[59]  Arnaud Gotlieb,et al.  Minimizing test suites in software product lines using weight-based genetic algorithms , 2013, GECCO '13.

[60]  Adam A. Porter,et al.  A history-based test prioritization technique for regression testing in resource constrained environments , 2002, ICSE '02.

[61]  Sergio A. Alvarez,et al.  Collaborative Recommendation via Adaptive Association Rule Mining , 2000 .

[62]  Gregg Rothermel,et al.  Test Case Prioritization: A Family of Empirical Studies , 2002, IEEE Trans. Software Eng..

[63]  Cem Kaner.  Improving the maintainability of automated test suites , 1997 .

[64]  Robert Feldt,et al.  Automated System Testing Using Visual GUI Testing Tools: A Comparative Study in Industry , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[65]  Usman Qamar,et al.  Rule induction using enhanced RIPPER algorithm for clinical decision support system , 2015, 2015 Sixth International Conference on Intelligent Control and Information Processing (ICICIP).

[66]  Waseem Shahzad,et al.  Feature subset selection using association rule mining and JRip classifier , 2013 .

[67]  Mark Harman,et al.  Transformed Vargha-Delaney Effect Size , 2015, SSBSE.

[68]  Austen Rainer,et al.  Case Study Research in Software Engineering - Guidelines and Examples , 2012 .

[69]  Tao Xie,et al.  To Be Optimal or Not in Test-Case Prioritization , 2016, IEEE Transactions on Software Engineering.

[70]  Gregg Rothermel,et al.  Prioritizing test cases for regression testing , 2000, ISSTA '00.

[71]  Claes Wohlin,et al.  Experimentation in software engineering: an introduction , 2000 .