Input-based adaptive randomized test case prioritization: A local beam search approach

We extend ART with the search-based algorithm for test case prioritization.Our techniques are as effective as the best search-based TCP technique.Our techniques are more efficient than genetic and greedy, but not ART.It is the first experiment to evaluate input-based search-based TCP techniques. Test case prioritization assigns the execution priorities of the test cases in a given test suite. Many existing test case prioritization techniques assume the full-fledged availability of code coverage data, fault history, or test specification, which are seldom well-maintained in real-world software development projects. This paper proposes a novel family of input-based local-beam-search adaptive-randomized techniques. They make adaptive tree-based randomized explorations with a randomized candidate test set strategy to even out the search space explorations among the branches of the exploration trees constructed by the test inputs in the test suite. We report a validation experiment on a suite of four medium-size benchmarks. The results show that our techniques achieve either higher APFD values than or the same mean APFD values as the existing code-coverage-based greedy or search-based prioritization techniques, including Genetic, Greedy and ART, in both our controlled experiment and case study. Our techniques are also significantly more efficient than the Genetic and Greedy, but are less efficient than ART.

[1]  Hareton K. N. Leung,et al.  A survey of code‐based change impact analysis techniques , 2013, Softw. Test. Verification Reliab..

[2]  Zhi Quan Zhou Using Coverage Information to Guide Test Case Selection in Adaptive Random Testing , 2010, 2010 IEEE 34th Annual Computer Software and Applications Conference Workshops.

[3]  Tsong Yueh Chen,et al.  An upper bound on software testing effectiveness , 2008, TSEM.

[4]  Steven Skiena,et al.  The Algorithm Design Manual , 2020, Texts in Computer Science.

[5]  John H. Holland,et al.  Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence , 1992 .

[6]  Mary Lou Soffa,et al.  TimeAware test suite prioritization , 2006, ISSTA '06.

[7]  Lionel C. Briand,et al.  Random Testing: Theoretical Results and Practical Implications , 2012, IEEE Transactions on Software Engineering.

[8]  Rajesh Subramanyan,et al.  Supporting Concern-Based Regression Testing and Prioritization in a Model-Driven Environment , 2010, 2010 IEEE 34th Annual Computer Software and Applications Conference Workshops.

[9]  Rajib Mall,et al.  Model-based regression test case prioritization , 2010, SOEN.

[10]  Gregg Rothermel,et al.  An empirical study of the effect of time constraints on the cost-benefits of regression testing , 2008, SIGSOFT '08/FSE-16.

[11]  Éva Tardos,et al.  Algorithm design , 2005 .

[12]  Gregg Rothermel,et al.  A Static Approach to Prioritizing JUnit Test Cases , 2012, IEEE Transactions on Software Engineering.

[13]  Wolfgang Grieskamp,et al.  Multi-paradigmatic Model-Based Testing , 2006, FATES/RV.

[14]  Nadine Mandran,et al.  Prioritizing test cases with string distances , 2011, Automated Software Engineering.

[15]  Bertrand Meyer,et al.  ARTOO , 2008, 2008 ACM/IEEE 30th International Conference on Software Engineering.

[16]  T. H. Tse,et al.  XML-manipulating test case prioritization for XML-manipulating services , 2011, J. Syst. Softw..

[17]  Bo Jiang,et al.  Prioritizing Test Cases for Regression Testing of Location-Based Services: Metrics, Techniques, and Case Study , 2014, IEEE Transactions on Services Computing.

[18]  Gregg Rothermel,et al.  Bridging the gap between the total and additional test-case prioritization strategies , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[19]  Baowen Xu,et al.  An empirical study on the effectiveness of time-aware test case prioritization techniques , 2011, SAC.

[20]  Lionel C. Briand,et al.  A practical guide for using statistical tests to assess randomized algorithms in software engineering , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[21]  Adam A. Porter,et al.  A history-based test prioritization technique for regression testing in resource constrained environments , 2002, ICSE '02.

[22]  Gregg Rothermel,et al.  Test Case Prioritization: A Family of Empirical Studies , 2002, IEEE Trans. Software Eng..

[23]  Bo Jiang,et al.  Bypassing Code Coverage Approximation Limitations via Effective Input-Based Randomized Test Case Prioritization , 2013, 2013 IEEE 37th Annual Computer Software and Applications Conference.

[24]  Shen Lin Computer solutions of the traveling salesman problem , 1965 .

[25]  Tsong Yueh Chen,et al.  Adaptive Random Testing: The ART of test case diversity , 2010, J. Syst. Softw..

[26]  Tsong Yueh Chen,et al.  Mirror adaptive random testing , 2004, Inf. Softw. Technol..

[27]  Ahmed E. Hassan,et al.  Static test case prioritization using topic models , 2014, Empirical Software Engineering.

[28]  Lefteris Angelis,et al.  An Empirical Study on Views of Importance of Change Impact Analysis Issues , 2008, IEEE Transactions on Software Engineering.

[29]  Wei-Tek Tsai,et al.  Regression testing in an industrial environment , 1998, CACM.

[30]  C. A. R. Hoare,et al.  Differential static analysis: opportunities, applications, and challenges , 2010, FoSER '10.

[31]  Harish Patil,et al.  Pin: building customized program analysis tools with dynamic instrumentation , 2005, PLDI '05.

[32]  Gregg Rothermel,et al.  A safe, efficient regression test selection technique , 1997, TSEM.

[33]  Mary Lou Soffa,et al.  A methodology for controlling the size of a test suite , 1993, TSEM.

[34]  Dan Gusfield Algorithms on Strings, Trees, and Sequences - Computer Science and Computational Biology , 1997 .

[35]  Bogdan Korel,et al.  Model-based test prioritization heuristic methods and their evaluation , 2007, A-MOST '07.

[36]  Bo Qu,et al.  Test Case Prioritization for Black Box Testing , 2007, 31st Annual International Computer Software and Applications Conference (COMPSAC 2007).

[37]  Mark Harman,et al.  Regression testing minimization, selection and prioritization: a survey , 2012, Softw. Test. Verification Reliab..

[38]  Gregory Tassey,et al.  Prepared for what , 2007 .

[39]  Mark Harman,et al.  A Theoretical and Empirical Study of Search-Based Testing: Local, Global, and Hybrid Search , 2010, IEEE Transactions on Software Engineering.

[40]  T. H. Tse,et al.  Adaptive Random Test Case Prioritization , 2009, 2009 IEEE/ACM International Conference on Automated Software Engineering.

[41]  Tao Xie,et al.  Time-aware test-case prioritization using integer linear programming , 2009, ISSTA.

[42]  Xiang Li,et al.  Assuring the model evolution of protocol software specifications by regression testing process improvement , 2011, Softw. Pract. Exp..

[43]  T. H. Tse,et al.  Preemptive Regression Testingof Workflow-Based Web Services , 2015, IEEE Transactions on Services Computing.

[44]  T. H. Tse,et al.  A Subsumption Hierarchy of Test Case Prioritization for Composite Services , 2015, IEEE Transactions on Services Computing.

[45]  Lionel C. Briand,et al.  Adaptive random testing: an illusion of effectiveness? , 2011, ISSTA '11.

[46]  Joseph Robert Horgan,et al.  A study of effective regression testing in practice , 1997, Proceedings The Eighth International Symposium on Software Reliability Engineering.

[47]  Zhenyu Chen,et al.  A Simulation Study on Some Search Algorithms for Regression Test Case Prioritization , 2010, 2010 10th International Conference on Quality Software.

[48]  Anne M. Denton,et al.  A clustering approach to improving test case prioritization: An industrial case study , 2011, 2011 27th IEEE International Conference on Software Maintenance (ICSM).

[49]  Mark Harman,et al.  Search Algorithms for Regression Test Case Prioritization , 2007, IEEE Transactions on Software Engineering.

[50]  David H. Wolpert,et al.  No free lunch theorems for optimization , 1997, IEEE Trans. Evol. Comput..

[51]  Gregg Rothermel,et al.  Selecting a Cost-Effective Test Case Prioritization Technique , 2004, Software Quality Journal.

[52]  Amitabh Srivastava,et al.  Effectively prioritizing tests in development environment , 2002, ISSTA '02.

[53]  Xu Zhao,et al.  Adaptive Test-Case Prioritization Guided by Output Inspection , 2013, 2013 IEEE 37th Annual Computer Software and Applications Conference.

[54]  Tsong Yueh Chen,et al.  Distribution Metric Driven Adaptive Random Testing , 2007 .

[55]  I. K. Mak,et al.  Adaptive Random Testing , 2004, ASIAN.

[56]  W. Chan,et al.  Preemptive Regression Testing of Workflow-based Web Services , 2014 .