Test case prioritization for object-oriented software: An adaptive random sequence approach based on clustering

Test case prioritization (TCP) attempts to improve fault detection effectiveness by scheduling the important test cases to be executed earlier, where the importance is determined by some criteria or strategies. Adaptive random sequences (ARSs) can be used to improve the effectiveness of TCP based on white-box information (such as code coverage information) or black-box information (such as test input information). To improve the testing effectiveness for object-oriented software in regression testing, in this paper, we present an ARS approach based on clustering techniques using black-box information. We use two clustering methods: (1) clustering test cases according to the number of objects and methods, using the K-means and K-medoids clustering algorithms; and (2) clustered based on an object and method invocation sequence similarity metric using the K-medoids clustering algorithm. Our approach can construct ARSs that attempt to make their neighboring test cases as diverse as possible. Experimental studies were also conducted to verify the proposed approach, with the results showing both enhanced probability of earlier fault detection, and higher effectiveness than random prioritization and method coverage TCP technique.

[1]  Bertrand Meyer,et al.  ARTOO , 2008, 2008 ACM/IEEE 30th International Conference on Software Engineering.

[2]  Huai Liu,et al.  How Effectively Does Metamorphic Testing Alleviate the Oracle Problem? , 2014, IEEE Transactions on Software Engineering.

[3]  Hareton K. N. Leung,et al.  Combinatorial testing, random testing, and adaptive random testing for detecting interaction triggered failures , 2015, Inf. Softw. Technol..

[4]  Huai Liu,et al.  An Application of Adaptive Random Sequence in Test Case Prioritization , 2014, SEKE.

[5]  Humera Tariq,et al.  K-Means Cluster Analysis for Image Segmentation , 2014 .

[6]  Qi Luo,et al.  A large-scale empirical comparison of static and dynamic test case prioritization techniques , 2016, SIGSOFT FSE.

[7]  R. Suganya,et al.  Data Mining Concepts and Techniques , 2010 .

[8]  I. K. Mak,et al.  Adaptive Random Testing , 2004, ASIAN.

[9]  Darko Marinov,et al.  An extensive study of static regression test selection in modern software evolution , 2016, SIGSOFT FSE.

[10]  Tsong Yueh Chen,et al.  An upper bound on software testing effectiveness , 2008, TSEM.

[11]  Tsong Yueh Chen,et al.  In black and white: an integrated approach to class-level testing of object-oriented programs , 1998, TSEM.

[12]  Nadine Mandran,et al.  Prioritizing test cases with string distances , 2011, Automated Software Engineering.

[13]  Richard Torkar,et al.  Empirical evaluations on the cost-effectiveness of state-based testing: An industrial case study , 2014, Inf. Softw. Technol..

[14]  Robert V. Binder,et al.  Testing object‐oriented software: a survey , 1996 .

[15]  Peter J. Rousseeuw,et al.  Clustering by means of medoids , 1987 .

[16]  Huai Liu,et al.  Code Coverage of Adaptive Random Testing , 2013, IEEE Transactions on Reliability.

[17]  Tsong Yueh Chen,et al.  In Black and White: An Integrated Approach to Object-Oriented Program Testing , 1998 .

[18]  Tsong Yueh Chen,et al.  Adaptive Random Testing: The ART of test case diversity , 2010, J. Syst. Softw..

[19]  Jinfu Chen,et al.  An Adaptive Sequence Approach for OOS Test Case Prioritization , 2016, 2016 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW).

[20]  Yves Le Traon,et al.  Comparing White-Box and Black-Box Test Prioritization , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE).

[21]  Xiaofang Zhang,et al.  Test Case Prioritization Using Adaptive Random Sequence with Category-Partition-Based Distance , 2016, 2016 IEEE International Conference on Software Quality, Reliability and Security (QRS).

[22]  Rui Abreu,et al.  Prioritizing tests for software fault diagnosis , 2011, Softw. Pract. Exp..

[23]  Huai Liu,et al.  A Cost-Effective Random Testing Method for Programs with Non-Numeric Inputs , 2016, IEEE Transactions on Computers.

[24]  A. Vargha,et al.  A Critique and Improvement of the CL Common Language Effect Size Statistics of McGraw and Wong , 2000 .

[25]  Yves Le Traon,et al.  Threats to the validity of mutation-based test assessment , 2016, ISSTA.

[26]  Hae-Sang Park,et al.  A simple and fast algorithm for K-medoids clustering , 2009, Expert Syst. Appl..

[27]  Mark Harman,et al.  Search Based Software Engineering: Techniques, Taxonomy, Tutorial , 2010, LASER Summer School.

[28]  Gregg Rothermel,et al.  Prioritizing test cases for regression testing , 2000, ISSTA '00.

[29]  Florin Gorunescu,et al.  Data Mining - Concepts, Models and Techniques , 2011, Intelligent Systems Reference Library.

[30]  Mauro Pezzè,et al.  Testing object-oriented software , 2004, Proceedings. 26th International Conference on Software Engineering.

[31]  Robert Tibshirani,et al.  Estimating the number of clusters in a data set via the gap statistic , 2000 .

[32]  Fei Rong Object-oriented Software Cluster-level Testing Based on UML Sequence Diagram , 2005 .

[33]  Jinfu Chen,et al.  A Similarity Metric for the Inputs of OO Programs and Its Application in Adaptive Random Testing , 2017, IEEE Transactions on Reliability.

[34]  Myra B. Cohen,et al.  An orchestrated survey of methodologies for automated software test case generation , 2013, J. Syst. Softw..

[35]  David Leon,et al.  Finding failures by cluster analysis of execution profiles , 2001, Proceedings of the 23rd International Conference on Software Engineering. ICSE 2001.

[36]  Lionel C. Briand,et al.  A Hitchhiker's guide to statistical tests for assessing randomized algorithms in software engineering , 2014, Softw. Test. Verification Reliab..

[37]  Mark Harman,et al.  Clustering test cases to achieve effective and scalable prioritisation incorporating expert knowledge , 2009, ISSTA.

[38]  Hui Chen,et al.  A literature survey on smart cities , 2015, Science China Information Sciences.

[39]  Gregg Rothermel,et al.  Test Case Prioritization: A Family of Empirical Studies , 2002, IEEE Trans. Software Eng..

[40]  Mark Harman,et al.  An Analysis and Survey of the Development of Mutation Testing , 2011, IEEE Transactions on Software Engineering.

[41]  R. Hamlet RANDOM TESTING , 1994 .

[42]  Dave Towey,et al.  A revisit of three studies related to random testing , 2015, Science China Information Sciences.