DART: directed automated random testing

We present a new tool, named DART, for automatically testing software that combines three main techniques: (1) automated extraction of the interface of a program with its external environment using static source-code parsing; (2) automatic generation of a test driver for this interface that performs random testing to simulate the most general environment the program can operate in; and (3) dynamic analysis of how the program behaves under random testing and automatic generation of new test inputs to direct systematically the execution along alternative program paths. Together, these three techniques constitute Directed Automated Random Testing, or DART for short. The main strength of DART is thus that testing can be performed completely automatically on any program that compiles -- there is no need to write any test driver or harness code. During testing, DART detects standard errors such as program crashes, assertion violations, and non-termination. Preliminary experiments to unit test several examples of C programs are very encouraging.

[1]  James C. King,et al.  Symbolic execution and program testing , 1976, CACM.

[2]  Murray Hill,et al.  Lint, a C Program Checker , 1978 .

[3]  Roger M. Needham,et al.  Using encryption for authentication in large networks of computers , 1978, CACM.

[4]  Glenford J. Myers,et al.  Art of Software Testing , 1979 .

[5]  Carlos Urias Munoz,et al.  Automatic Generation of Random Self-Checking Test Cases , 1983, IBM Syst. J..

[6]  Bogdan Korel,et al.  A dynamic approach of test data generation , 1990, Proceedings. Conference on Software Maintenance 1990.

[7]  Robert O. Hastings,et al.  Fast detection of memory leaks and access errors , 1991 .

[8]  Gavin Lowe,et al.  An Attack on the Needham-Schroeder Public-Key Authentication Protocol , 1995, Inf. Process. Lett..

[9]  A. Jefferson Offutt,et al.  A semantic model of program faults , 1996, ISSTA '96.

[10]  Gavin Lowe,et al.  Breaking and Fixing the Needham-Schroeder Public-Key Protocol Using FDR , 1996, Softw. Concepts Tools.

[11]  Patrice Godefroid,et al.  Model checking for programming languages using VeriSoft , 1997, POPL '97.

[12]  Patrice Godefroid,et al.  Automatically closing open reactive programs , 1998, PLDI.

[13]  Mary Lou Soffa,et al.  Generating test data for branch coverage , 2000, Proceedings ASE 2000. Fifteenth IEEE International Conference on Automated Software Engineering.

[14]  Barton P. Miller,et al.  An empirical study of the robustness of Windows NT applications using random testing , 2000 .

[15]  William R. Bush,et al.  A static analyzer for finding dynamic programming errors , 2000 .

[16]  K. Claessen,et al.  QuickCheck: a lightweight tool for random testing of Haskell programs , 2000, ICFP '00.

[17]  William R. Bush,et al.  A static analyzer for finding dynamic programming errors , 2000, Softw. Pract. Exp..

[18]  Koen Claessen,et al.  QuickCheck: a lightweight tool for random testing of Haskell programs , 2000, ACM SIGPLAN International Conference on Functional Programming.

[19]  Anna Philippou,et al.  Tools and Algorithms for the Construction and Analysis of Systems , 2018, Lecture Notes in Computer Science.

[20]  Sriram K. Rajamani,et al.  The SLAM Toolkit , 2001, CAV.

[21]  Sarfraz Khurshid,et al.  Korat: automated testing based on Java predicates , 2002, ISSTA '02.

[22]  George C. Necula,et al.  CCured: type-safe retrofitting of legacy code , 2002, POPL '02.

[23]  Scott D. Stoller.  Domain partitioning for open reactive systems , 2002, ISSTA '02.

[24]  Dawson R. Engler,et al.  A system and language for building system-specific, static analyses , 2002, PLDI '02.

[25]  George C. Necula,et al.  CIL: Intermediate Language and Tools for Analysis and Transformation of C Programs , 2002, CC.

[26]  Thomas A. Henzinger,et al.  Lazy abstraction , 2002, POPL '02.

[27]  G. Tassey.  The economic impacts of inadequate infrastructure for software testing , 2002 .

[28]  Monica S. Lam,et al.  Automatic extraction of object-oriented component interfaces , 2002, ISSTA '02.

[29]  Jon Edvardsson,et al.  A Survey on Automatic Test Data Generation , 2002 .

[30]  Sarfraz Khurshid,et al.  Test input generation with java PathFinder , 2004, ISSTA '04.

[31]  Thomas A. Henzinger,et al.  Generating tests from counterexamples , 2004, Proceedings. 26th International Conference on Software Engineering.

[32]  Sarfraz Khurshid,et al.  Exploring very large state spaces using genetic algorithms , 2004, International Journal on Software Tools for Technology Transfer.

[33]  David Notkin,et al.  Symstra: A Framework for Generating Object-Oriented Unit Tests Using Symbolic Execution , 2005, TACAS.

[34]  Michael D. Ernst,et al.  Continuous testing in Eclipse , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[35]  C. Csallner,et al.  Check 'n' crash: combining static checking and testing , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[36]  Sumit Gulwani,et al.  Precise interprocedural analysis using random interpretation , 2005, POPL '05.

[37]  Pavol Cerný,et al.  Synthesis of interface specifications for Java classes , 2005, POPL '05.