Adaptive Random Test Case Prioritization

Regression testing assures changed programs against unintended amendments. Rearranging the execution order of test cases is a key idea to improve their effectiveness. Paradoxically, many test case prioritization techniques resolve tie cases using the random selection approach, and yet random ordering of test cases has been considered as ineffective. Existing unit testing research unveils that adaptive random testing (ART) is a promising candidate that may replace random testing (RT). In this paper, we not only propose a new family of coverage-based ART techniques, but also show empirically that they are statistically superior to the RT-based technique in detecting faults. Furthermore, one of the ART prioritization techniques is consistently comparable to some of the best coverage-based prioritization techniques (namely, the "additional" techniques) and yet involves much less time cost.

[1]  Rajiv Gupta,et al.  A methodology for controlling the size of a test suite , 1990, Proceedings. Conference on Software Maintenance 1990.

[2]  Thomas J. Ostrand,et al.  Experiments on the effectiveness of dataflow- and control-flow-based test adequacy criteria , 1994, Proceedings of 16th International Conference on Software Engineering.

[3]  Joseph Robert Horgan,et al.  A study of effective regression testing in practice , 1997, Proceedings The Eighth International Symposium on Software Reliability Engineering.

[4]  Gregg Rothermel,et al.  A safe, efficient regression test selection technique , 1997, TSEM.

[5]  Wei-Tek Tsai,et al.  Regression testing in an industrial environment , 1998, CACM.

[6]  Gregg Rothermel,et al.  Prioritizing test cases for regression testing , 2000, ISSTA '00.

[7]  Amitabh Srivastava,et al.  Effectively prioritizing tests in development environment , 2002, ISSTA '02.

[8]  Gregg Rothermel,et al.  Test Case Prioritization: A Family of Empirical Studies , 2002, IEEE Trans. Software Eng..

[9]  Dave Towey,et al.  Restricted Random Testing , 2002, ECSQ.

[10]  Gregg Rothermel,et al.  Selecting a Cost-Effective Test Case Prioritization Technique , 2004, Software Quality Journal.

[11]  Gregg Rothermel,et al.  On test suite composition and cost-effective regression testing , 2004, TSEM.

[12]  Gregg Rothermel,et al.  Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and its Potential Impact , 2005, Empirical Software Engineering.

[13]  Gregg Rothermel,et al.  Prioritizing JUnit Test Cases: An Empirical Assessment and Cost-Benefits Analysis , 2006, Empirical Software Engineering.

[14]  Bertrand Meyer,et al.  Object distance and its application to adaptive random testing of object-oriented programs , 2006, RT '06.

[15]  Mary Lou Soffa,et al.  TimeAware test suite prioritization , 2006, ISSTA '06.

[16]  Koushik Sen,et al.  Concolic testing , 2007, ASE.

[17]  Bertrand Meyer,et al.  Experimental assessment of random testing for object-oriented software , 2007, ISSTA '07.

[18]  Mark Harman,et al.  Search Algorithms for Regression Test Case Prioritization , 2007, IEEE Transactions on Software Engineering.

[19]  Tsong Yueh Chen,et al.  Quasi-Random Testing , 2005, IEEE Transactions on Reliability.

[20]  Suresh Jagannathan,et al.  PHALANX: a graph-theoretic framework for test case prioritization , 2008, SAC '08.

[21]  Bertrand Meyer,et al.  ARTOO: adaptive random testing for object-oriented software , 2008, ICSE.

[22]  T. Y. Chen,et al.  Adaptive Random Testing , 2004, ASIAN.

[23]  Tsong Yueh Chen,et al.  An upper bound on software testing effectiveness , 2008, TSEM.

[24]  Tsong Yueh Chen,et al.  How Well Do Test Case Prioritization Techniques Support Statistical Fault Localization , 2009, 2009 33rd Annual IEEE International Computer Software and Applications Conference.

[25]  T. H. Tse,et al.  Tag-Based Techniques for Black-Box Test Case Prioritization for Service Testing , 2009, 2009 Ninth International Conference on Quality Software.

[26]  T. H. Tse,et al.  Test case prioritization for regression testing of service-oriented business applications , 2009, WWW '09.

[27]  Tsong Yueh Chen,et al.  Adaptive Random Testing: The ART of test case diversity , 2010, J. Syst. Softw..