Kernel Density Adaptive Random Testing

Mutation analysis is used to assess the effectiveness of a test data generation technique at finding faults. Once a mutant is killed, decisions must be made whether to diversify or intensify the subsequent test inputs. Diversification employs a wide range of test inputs with the aim of increasing the chances of killing new mutants. By contrast, intensification selects test inputs which are similar to those previously shown to be successful, taking advantage of overlaps in the conditions under which mutants can be killed. This paper explores the trade-off between diversification and intensification by augmenting Adaptive Random Testing (ART) to estimate the Kernel Density (KD-ART) of input values which are found to kill mutants. The results suggest that intensification is typically more effective at finding faults than diversification. KD-ART (intensify) achieves 7.24% higher mutation score on average than KD-ART (diversify). Moreover, KD-ART is computationally less expensive than ART. The new technique requires an average 5.98% of the time taken before.

[1]  Huai Liu,et al.  Distributing test cases more evenly in adaptive random testing , 2008, J. Syst. Softw..

[2]  Huai Liu,et al.  Comparison of adaptive random testing and random testing under various testing and debugging scenarios , 2012, Softw. Pract. Exp..

[3]  Richard A. Davis,et al.  Remarks on Some Nonparametric Estimates of a Density Function , 2011 .

[4]  Johannes Mayer,et al.  Lattice-based adaptive random testing , 2005, ASE.

[5]  Andreas Zeller,et al.  Mutation-Driven Generation of Unit Tests and Oracles , 2010, IEEE Transactions on Software Engineering.

[6]  Huai Liu,et al.  Adaptive random testing based on distribution metrics , 2009, J. Syst. Softw..

[7]  James Miller,et al.  A Novel Evolutionary Approach for Adaptive Random Testing , 2009, IEEE Transactions on Reliability.

[8]  Xiaodong Xie,et al.  Adaptive Random Test Case Generation for Combinatorial Testing , 2012, 2012 IEEE 36th Annual Computer Software and Applications Conference.

[9]  Mike Papadakis,et al.  An Empirical Evaluation of the First and Second Order Mutation Testing Strategies , 2010, 2010 Third International Conference on Software Testing, Verification, and Validation Workshops.

[10]  Mark Harman,et al.  An Analysis and Survey of the Development of Mutation Testing , 2011, IEEE Transactions on Software Engineering.

[11]  Tsong Yueh Chen,et al.  Adaptive Random Testing: The ART of test case diversity , 2010, J. Syst. Softw..

[12]  Giuliano Antoniol,et al.  Automatic mutation test input data generation via ant colony , 2007, GECCO '07.

[13]  Lionel C. Briand,et al.  Random Testing: Theoretical Results and Practical Implications , 2012, IEEE Transactions on Software Engineering.

[14]  Gregg Rothermel,et al.  Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and its Potential Impact , 2005, Empirical Software Engineering.

[15]  Mark Harman,et al.  A study of equivalent and stubborn mutation operators using human analysis of equivalence , 2014, ICSE.

[16]  F. A. Seiler,et al.  Numerical Recipes in C: The Art of Scientific Computing , 1989 .

[17]  Simeon C. Ntafos,et al.  An Evaluation of Random Testing , 1984, IEEE Transactions on Software Engineering.

[18]  Tsong Yueh Chen,et al.  Enhanced lattice-based adaptive random testing , 2009, SAC '09.

[19]  Johannes Mayer,et al.  Towards the determination of typical failure patterns , 2007, SOQUA '07.

[20]  Yue Jia,et al.  MILU: A Customizable, Runtime-Optimized Higher Order Mutation Testing Tool for the Full C Language , 2008, Testing: Academic & Industrial Conference - Practice and Research Techniques (taic part 2008).

[21]  M. Rosenblatt Remarks on Some Nonparametric Estimates of a Density Function , 1956 .

[22]  Ki-Hyun Jung,et al.  Normalized Adaptive Random Test for Integration Tests , 2010, 2010 IEEE 34th Annual Computer Software and Applications Conference Workshops.

[23]  Yves Le Traon,et al.  Trivial Compiler Equivalence: A Large Scale Empirical Study of a Simple, Fast and Effective Equivalent Mutant Detection Technique , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[24]  Lee J. White,et al.  A Domain Strategy for Computer Program Testing , 1980, IEEE Transactions on Software Engineering.

[25]  Koushik Sen,et al.  CUTE: a concolic unit testing engine for C , 2005, ESEC/FSE-13.

[26]  Mark Harman,et al.  Strong higher order mutation-based test data generation , 2011, ESEC/FSE '11.

[27]  Fei-Ching Kuo An Indepth Study of Mirror Adaptive Random Testing , 2009, 2009 Ninth International Conference on Quality Software.

[28]  Gregg Rothermel,et al.  An experimental determination of sufficient mutant operators , 1996, TSEM.

[29]  Jinfu Chen,et al.  Adaptive random prioritization for interaction test suites , 2014, SAC.

[30]  John A. Clark,et al.  Subdomain-based test data generation , 2015, J. Syst. Softw..

[31]  E. Parzen On Estimation of a Probability Density Function and Mode , 1962 .

[32]  A. Jefferson Offutt,et al.  Constraint-Based Automatic Test Data Generation , 1991, IEEE Trans. Software Eng..

[33]  Tsong Yueh Chen,et al.  The ART of Divide and Conquer: An Innovative Approach to Improving the Efficiency of Adaptive Random Testing , 2013, 2013 13th International Conference on Quality Software.

[34]  Phil McMinn,et al.  Search-Based Software Testing: Past, Present and Future , 2011, 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops.

[35]  Koushik Sen,et al.  DART: directed automated random testing , 2005, PLDI '05.

[36]  Tsong Yueh Chen,et al.  An upper bound on software testing effectiveness , 2008, TSEM.

[37]  Xin-She Yang,et al.  Metaheuristic Optimization: Algorithm Analysis and Open Problems , 2011, SEA.

[38]  Huai Liu,et al.  Code Coverage of Adaptive Random Testing , 2013, IEEE Transactions on Reliability.

[39]  Lionel C. Briand,et al.  Adaptive random testing: an illusion of effectiveness? , 2011, ISSTA '11.

[40]  Lionel C. Briand,et al.  Is mutation an appropriate tool for testing experiments? , 2005, ICSE.

[41]  Tsong Yueh Chen,et al.  Adaptive Random Testing , 2004, ASIAN.