Tool support for randomized unit testing

There are several problem areas that must be addressed when applying randomization to unit testing. As yet no general, fully automated solution that works for all units has been proposed. We therefore have developed RUTE-J, a Java package intended to help programmers do randomized unit testing in Java. In this paper, we describe RUTE-J and illustrate how it supports the development of per-unit solutions for the problems of randomized unit testing. We report on an experiment in which we applied RUTE-J to the standard Java TreeMap class, measuring the efficiency and effectiveness of the technique. We also illustrate the use of randomized testing in experimentation, by adapting RUTE-J so that it generates randomized minimal covering test suites, and measuring the effectiveness of the test suites generated.

[1]  James H. Andrews,et al.  General Test Result Checking with Log File Analysis , 2003, IEEE Trans. Software Eng..

[2]  Yi Sun,et al.  Automated generation of test programs from closed specifications of classes and test cases , 2004, Proceedings. 26th International Conference on Software Engineering.

[3]  Sarfraz Khurshid,et al.  Korat: automated testing based on Java predicates , 2002, ISSTA '02.

[4]  Richard G. Hamlet,et al.  Partition Testing Does Not Inspire Confidence , 1990, IEEE Trans. Software Eng..

[5]  Catherine Oriat,et al.  Jartege: A Tool for Random Generation of Unit Tests for Java Classes , 2004, QoSA/SOQUA.

[6]  W. Masri,et al.  An empirical evaluation of test case filtering techniques based on exercising complex information flows , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[7]  Kent L. Beck,et al.  Test-driven Development - by example , 2002, The Addison-Wesley signature series.

[8]  Lydie du Bousquet,et al.  Filtering TOBIAS Combinatorial Test Suites , 2004, FASE.

[9]  Thomas D. LaToza,et al.  Software Development at Microsoft Observed , 2005 .

[10]  Nikolai Tillmann,et al.  Parameterized unit tests , 2005, ESEC/FSE-13.

[11]  Koushik Sen,et al.  CUTE: a concolic unit testing engine for C , 2005, ESEC/FSE-13.

[12]  Lionel C. Briand,et al.  Is mutation an appropriate tool for testing experiments? , 2005, ICSE.

[13]  Phyllis G. Frankl,et al.  The ASTOOT approach to testing object-oriented programs , 1994, TSEM.

[14]  Bev Littlewood,et al.  Evaluating Testing Methods by Delivered Reliability , 1998, IEEE Trans. Software Eng..

[15]  Gregg Rothermel,et al.  An empirical study of the effects of minimization on the fault detection capabilities of test suites , 1998, Proceedings. International Conference on Software Maintenance (Cat. No. 98CB36272).

[16]  Sarfraz Khurshid,et al.  Test input generation with java PathFinder , 2004, ISSTA '04.

[17]  Walter J. Gutjahr,et al.  Partition Testing vs. Random Testing: The Influence of Uncertainty , 1999, IEEE Trans. Software Eng..

[18]  Gregg Rothermel,et al.  A controlled experiment assessing test case prioritization techniques via mutation faults , 2005, 21st IEEE International Conference on Software Maintenance (ICSM'05).

[19]  Richard G. Hamlet,et al.  Automatically Checking an Implementation against Its Formal Specification , 2000, IEEE Trans. Software Eng..

[20]  Mark Hennessy,et al.  An analysis of rule coverage as a criterion in generating minimal test suites for grammar-based software , 2005, ASE.

[21]  David Lee,et al.  Principles and methods of testing finite state machines-a survey , 1996, Proc. IEEE.

[22]  Klaus Havelund,et al.  Model checking JAVA programs using JAVA PathFinder , 2000, International Journal on Software Tools for Technology Transfer.

[23]  Barton P. Miller,et al.  An empirical study of the reliability of UNIX utilities , 1990, Commun. ACM.

[24]  Daniel Hoffman,et al.  ClassBench: a framework for automated class testing , 1997 .

[25]  Nikolai Tillmann,et al.  Parameterized unit tests with unit meister , 2005, ESEC/FSE-13.

[26]  Phyllis G. Frankl,et al.  An Experimental Comparison of the Effectiveness of Branch Testing and Data Flow Testing , 1993, IEEE Trans. Software Eng..

[27]  Yong Lei,et al.  Minimization of randomized unit test cases , 2005, 16th IEEE International Symposium on Software Reliability Engineering (ISSRE'05).

[28]  A. Jefferson Offutt,et al.  Mutation 2000: uniting the orthogonal , 2001 .

[29]  Koen Claessen,et al.  QuickCheck: a lightweight tool for random testing of Haskell programs , 2000, ICFP.

[30]  Daniel Hoffman,et al.  ClassBench: A Framework for Automated Class Testing , 1997, Softw. Pract. Exp..

[31]  Michael D. Ernst,et al.  An overview of JML tools and applications , 2003, International Journal on Software Tools for Technology Transfer.

[32]  Yannis Smaragdakis,et al.  JCrasher: an automatic robustness tester for Java , 2004, Softw. Pract. Exp..

[33]  W. M. McKeeman,et al.  Differential Testing for Software , 1998, Digit. Tech. J..

[34]  Sarfraz Khurshid,et al.  TestEra: Specification-Based Testing of Java Programs Using SAT , 2004, Automated Software Engineering.

[35]  Andreas Zeller,et al.  Simplifying and Isolating Failure-Inducing Input , 2002, IEEE Trans. Software Eng..