Hybrid Concolic Testing

We present hybrid concolic testing, an algorithm that interleaves random testing with concolic execution to obtain both a deep and a wide exploration of program state space. Our algorithm generates test inputs automatically by interleaving random testing until saturation with bounded exhaustive symbolic exploration of program points. It thus combines the ability of random search to reach deep program states quickly together with the ability of concolic testing to explore states in a neighborhood exhaustively. We have implemented our algorithm on top of CUTE and applied it to obtain better branch coverage for an editor implementation (VIM 5.7, 150 K lines of code) as well as a data structure implementation in C. Our experiments suggest that hybrid concolic testing can handle large programs and provide, for the same testing budget, almost 4times the branch coverage than random testing and almost 2times that of concolic testing.

[1]  James C. King,et al.  Symbolic execution and program testing , 1976, CACM.

[2]  Lori A. Clarke,et al.  A System to Generate Test Data and Symbolically Execute Programs , 1976, IEEE Transactions on Software Engineering.

[3]  J. J. Whelan,et al.  5th international conference on software engineering , 1981, SOEN.

[4]  Carlos Urias Munoz,et al.  Automatic Generation of Random Self-Checking Test Cases , 1983, IBM Syst. J..

[5]  A. Jefferson Offutt,et al.  A semantic model of program faults , 1996, ISSTA '96.

[6]  John C. Mitchell,et al.  Foundations for programming languages , 1996, Foundation of computing series.

[7]  Bogdan Korel,et al.  The chaining approach for software test data generation , 1996, TSEM.

[8]  Mary Lou Soffa,et al.  Automated test data generation using an iterative relaxation method , 1998, SIGSOFT '98/FSE-6.

[9]  Adnan Aziz,et al.  Enhancing simulation with BDDs and ATPG , 1999, DAC '99.

[10]  Barton P. Miller,et al.  An empirical study of the robustness of Windows NT applications using random testing , 2000 .

[11]  K. Claessen,et al.  QuickCheck: a lightweight tool for random testing of Haskell programs , 2000, ICFP '00.

[12]  Jiang Long,et al.  Smart simulation using collaborative formal and simulation engines , 2000, IEEE/ACM International Conference on Computer Aided Design. ICCAD - 2000. IEEE/ACM Digest of Technical Papers (Cat. No.00CH37140).

[13]  Neelam Gupta,et al.  Generating test data for functions with pointer inputs , 2002, Proceedings 17th IEEE International Conference on Automated Software Engineering,.

[14]  Thomas Ball,et al.  Abstraction-guided Test Generation: A Case Study , 2003 .

[15]  Sarfraz Khurshid,et al.  Test input generation with java PathFinder , 2004, ISSTA '04.

[16]  Thomas A. Henzinger,et al.  Generating tests from counterexamples , 2004, Proceedings. 26th International Conference on Software Engineering.

[17]  Yannis Smaragdakis,et al.  JCrasher: an automatic robustness tester for Java , 2004, Softw. Pract. Exp..

[18]  D. Notkin,et al.  Rostra: a framework for detecting redundant object-oriented unit tests , 2004, Proceedings. 19th International Conference on Automated Software Engineering, 2004..

[19]  Leonardo Mendonça de Moura,et al.  Generating efficient test sets with a model checker , 2004, Proceedings of the Second International Conference on Software Engineering and Formal Methods, 2004. SEFM 2004..

[20]  Nikolai Tillmann,et al.  Parameterized unit tests , 2005, ESEC/FSE-13.

[21]  Koushik Sen,et al.  CUTE: a concolic unit testing engine for C , 2005, ESEC/FSE-13.

[22]  David Notkin,et al.  Symstra: A Framework for Generating Object-Oriented Unit Tests Using Symbolic Execution , 2005, TACAS.

[23]  Donglin Liang,et al.  Coverage-directed test generation with model checkers: challenges and opportunities , 2005, 29th Annual International Computer Software and Applications Conference (COMPSAC'05).

[24]  Rupak Majumdar,et al.  Path slicing , 2005, PLDI '05.

[25]  Dawson R. Engler,et al.  Execution Generated Test Cases: How to Make Systems Code Crash Itself , 2005, SPIN.

[26]  C. Csallner,et al.  Check 'n' crash: combining static checking and testing , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[27]  Koushik Sen,et al.  DART: directed automated random testing , 2005, PLDI '05.

[28]  Yong Lei,et al.  Minimization of randomized unit test cases , 2005, 16th IEEE International Symposium on Software Reliability Engineering (ISSRE'05).

[29]  Songtao Xia,et al.  Automated test generation for engineering applications , 2005, ASE '05.

[30]  Corina S. Pasareanu,et al.  Test input generation for red-black trees using abstraction , 2005, ASE.

[31]  Michael D. Ernst,et al.  Eclat: Automatic Generation and Classification of Test Inputs , 2005, ECOOP.

[32]  Yannis Smaragdakis,et al.  Check 'n' crash: combining static checking and testing , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[33]  Marcelo d'Amorim,et al.  An Empirical Comparison of Automated Generation and Classification Techniques for Object-Oriented Unit Testing , 2006, 21st IEEE/ACM International Conference on Automated Software Engineering (ASE'06).