Using random test selection to gain confidence in modified software

This paper presents a method that addresses two practical issues concerning the use of random test selection for regression testing: the number of random samples needed from the test suite to provide reliable results, and the confidence levels of the predictions made by the random samples. The method applies the Chernoff bound, which has been applied in various randomized algorithms, to compute the error bound for random test selection. The paper presents three example applications, based on the method, for regression testing. The main benefits of the method are that it requires no distribution information about the test suite from which the samples are taken, and the computation of the confidence level is independent of the size of the test suite. The paper also presents the results of an empirical evaluation of the technique on a set of C programs, which have been used in many testing experiments, along with three of the GCC compilers. The results demonstrate the effectiveness of the method and show its potential for regression testing on real-world, large-scale applications.

[1]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[2]  Mary Jean Harrold,et al.  Regression test selection on system requirements , 2008, ISEC '08.

[3]  Robert Brownlie,et al.  Robust testing of AT&T PMX/StarMAIL using OATS , 1992, AT&T Technical Journal.

[4]  Rajeev Motwani,et al.  Randomized Algorithms , 1995, SIGA.

[5]  Josef Kittler,et al.  Pattern recognition : a statistical approach , 1982 .

[6]  Michael D. Ernst,et al.  Feedback-Directed Random Test Generation , 2007, 29th International Conference on Software Engineering (ICSE'07).

[7]  Thomas J. Ostrand,et al.  Experiments on the effectiveness of dataflow- and control-flow-based test adequacy criteria , 1994, Proceedings of 16th International Conference on Software Engineering.

[8]  Johannes Mayer,et al.  Lattice-based adaptive random testing , 2005, ASE.

[9]  Tsong Yueh Chen,et al.  Quasi-Random Testing , 2005, IEEE Transactions on Reliability.

[10]  Bertrand Meyer,et al.  On the Predictability of Random Tests for Object-Oriented Software , 2008, 2008 1st International Conference on Software Testing, Verification, and Validation.

[11]  Phyllis G. Frankl,et al.  Pythia: a regression test selection tool based on textual differencing , 1997 .

[12]  Alessandro Orso,et al.  Scaling regression testing to large software systems , 2004, SIGSOFT '04/FSE-12.

[13]  Phyllis G. Frankl,et al.  Empirical evaluation of the textual differencing regression testing technique , 1998, Proceedings. International Conference on Software Maintenance (Cat. No. 98CB36272).

[14]  Sarfraz Khurshid,et al.  Korat: automated testing based on Java predicates , 2002, ISSTA '02.

[15]  Elaine J. Weyuker,et al.  Analyzing Partition Testing Strategies , 1991, IEEE Trans. Software Eng..

[16]  Gregg Rothermel,et al.  A safe, efficient regression test selection technique , 1997, TSEM.

[17]  Michael D. Ernst,et al.  Eclat: Automatic Generation and Classification of Test Inputs , 2005, ECOOP.

[18]  David S. Rosenblum,et al.  TESTTUBE: a system for selective regression testing , 1994, Proceedings of 16th International Conference on Software Engineering.

[19]  Koushik Sen.  DART: Directed Automated Random Testing , 2009, Haifa Verification Conference.

[20]  David Hinkley,et al.  Bootstrap Methods: Another Look at the Jackknife , 2008 .

[21]  Dick Hamlet.  When only random testing will do , 2006, RT '06.

[22]  Bertrand Meyer,et al.  ARTOO: adaptive random testing for object-oriented software , 2008, ICSE.

[23]  Rupak Majumdar,et al.  Hybrid Concolic Testing , 2007, 29th International Conference on Software Engineering (ICSE'07).

[24]  Richard Torkar,et al.  New quality estimations in random testing , 2003, 14th International Symposium on Software Reliability Engineering, 2003. ISSRE 2003..

[25]  Tsong Yueh Chen,et al.  On the Expected Number of Failures Detected by Subdomain Testing and Random Testing , 1996, IEEE Trans. Software Eng..

[26]  Gadiel Seroussi,et al.  Vector sets for exhaustive testing of logic circuits , 1988, IEEE Trans. Inf. Theory.

[27]  Simeon C. Ntafos,et al.  An Evaluation of Random Testing , 1984, IEEE Transactions on Software Engineering.

[28]  Phyllis G. Frankl,et al.  Comparison of delivered reliability of branch, data flow and operational testing: A case study , 2000, ISSTA '00.

[29]  Koushik Sen,et al.  CUTE: a concolic unit testing engine for C , 2005, ESEC/FSE-13.

[30]  R. Hamlet.  RANDOM TESTING , 1994 .