Minimization of randomized unit test cases

We describe a framework for randomized unit testing, and give empirical evidence that generating unit test cases randomly and then minimizing the failing test cases results in significant benefits. Randomized generation of unit test cases (sequences of method calls) has been shown to allow high coverage and to be highly effective. However, failing test cases, if found, are often very long sequences of method calls. We show that Zeller and Hildebrandt's test case minimization algorithm significantly reduces the length of these sequences. We study the resulting benefits qualitatively and quantitatively, via a case study on found open-source data structures and an experiment on lab-built data structures

[1]  Catherine Oriat,et al.  Jartege: A Tool for Random Generation of Unit Tests for Java Classes , 2004, QoSA/SOQUA.

[2]  Robert F. Roggio,et al.  An automated oracle for software testing , 1992 .

[3]  Yi Sun,et al.  Automated generation of test programs from closed specifications of classes and test cases , 2004, Proceedings. 26th International Conference on Software Engineering.

[4]  James H. Andrews.  A case study of coverage-checked random data structure testing , 2004, Proceedings. 19th International Conference on Automated Software Engineering, 2004..

[5]  Lydie du Bousquet,et al.  Filtering TOBIAS Combinatorial Test Suites , 2004, FASE.

[6]  Koushik Sen,et al.  DART: directed automated random testing , 2005, PLDI '05.

[7]  Gary McGraw,et al.  Generating Software Test Data by Evolution , 2001, IEEE Trans. Software Eng..

[8]  Simeon C. Ntafos,et al.  An Evaluation of Random Testing , 1984, IEEE Transactions on Software Engineering.

[9]  Guan Huang.  Learning and initial use of a software testing technology: an exploratory study , 2004, ICSE 2004.

[10]  Lionel C. Briand,et al.  Is mutation an appropriate tool for testing experiments? , 2005, ICSE.

[11]  J.H. Andrews,et al.  Is mutation an appropriate tool for testing experiments? [software testing] , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[12]  Tsong Yueh Chen,et al.  TACCLE: a methodology for object-oriented software testing at the class and cluster levels , 2001, TSEM.

[13]  Siau-Cheng Khoo,et al.  Heuristic search with reachability tests for automated generation of test programs , 2004, Proceedings. 19th International Conference on Automated Software Engineering, 2004..

[14]  Matthew B. Dwyer,et al.  Automated environment generation for software model checking , 2003, 18th IEEE International Conference on Automated Software Engineering, 2003. Proceedings..

[15]  Elaine J. Weyuker,et al.  On Testing Non-Testable Programs , 1982, Comput. J..

[16]  W. M. McKeeman,et al.  Differential Testing for Software , 1998, Digit. Tech. J..

[17]  Yingjun Zhang,et al.  Broad-spectrum studies of log file analysis , 2000, Proceedings of the 2000 International Conference on Software Engineering. ICSE 2000 the New Millennium.

[18]  Glenford J. Myers,et al.  Art of Software Testing , 1979 .

[19]  Siau-Cheng Khoo,et al.  Heuristic search with reachability tests for automated generation of test programs , 2004 .

[20]  Cem Kaner,et al.  Testing Computer Software , 1988 .

[21]  Phyllis G. Frankl,et al.  The ASTOOT approach to testing object-oriented programs , 1994, TSEM.

[22]  David Lorge Parnas,et al.  Using Test Oracles Generated from Program Documentation , 1998, IEEE Trans. Software Eng..

[23]  Koen Claessen,et al.  QuickCheck: a lightweight tool for random testing of Haskell programs , 2000, ICFP.

[24]  Debra J. Richardson,et al.  Specification-based test oracles for reactive systems , 1992, International Conference on Software Engineering.

[25]  A. Jefferson Offutt,et al.  Mutation 2000: uniting the orthogonal , 2001 .

[26]  ZellerAndreas,et al.  Simplifying and Isolating Failure-Inducing Input , 2002 .

[27]  James H. Andrews,et al.  General Test Result Checking with Log File Analysis , 2003, IEEE Trans. Software Eng..