Automated unit testing of large industrial embedded software using concolic testing

Current testing practice in industry is often ineffective and slow to detect bugs, since most projects utilize manually generated test cases. Concolic testing alleviates this problem by automatically generating test cases that achieve high coverage. However, specialized execution platforms and resource constraints of embedded software hinder application of concolic testing to embedded software. To overcome these limitations, we have developed CONcrete and symBOLic (CONBOL) testing framework to unit test large size industrial embedded software automatically. To address the aforementioned limitations, CONBOL tests target units on a host PC platform by generating symbolic unit testing drivers/stubs automatically and applying heuristics to reduce false alarms caused by the imprecise drivers/stubs. We have applied CONBOL to four million lines long industrial embedded software and detected 24 new crash bugs. Furthermore, the development team of the target software adopted CONBOL to their development process to apply CONBOL to the revised target software regularly.

[1]  Moonzoo Kim,et al.  Industrial Application of Concolic Testing on Embedded Software: Case Studies , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[2]  Yannis Smaragdakis,et al.  JCrasher: an automatic robustness tester for Java , 2004, Softw. Pract. Exp..

[3]  C. Csallner,et al.  Check 'n' crash: combining static checking and testing , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[4]  Koushik Sen,et al.  Heuristics for Scalable Dynamic Test Generation , 2008, 2008 23rd IEEE/ACM International Conference on Automated Software Engineering.

[5]  Stephen McCamant,et al.  Loop-extended symbolic execution on binary programs , 2009, ISSTA.

[6]  Moonzoo Kim,et al.  Industrial application of concolic testing approach: A case study on libexif by using CREST-BV and KLEE , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[7]  Sarfraz Khurshid,et al.  Test input generation with java PathFinder , 2004, ISSTA '04.

[8]  Nikolai Tillmann,et al.  Environment Modeling for Automated Testing of Cloud Applications , 2011, CloudCom 2011.

[9]  Moonzoo Kim,et al.  A Comparative Study of Software Model Checkers as Unit Testing Tools: An Industrial Case Study , 2011, IEEE Transactions on Software Engineering.

[10]  Roy Osherove.  The Art of Unit Testing: With Examples in .NET , 2009 .

[11]  Patrice Godefroid,et al.  Automated Whitebox Fuzz Testing , 2008, NDSS.

[12]  Thomas A. Henzinger,et al.  Generating tests from counterexamples , 2004, Proceedings. 26th International Conference on Software Engineering.

[13]  Patrice Godefroid,et al.  Precise pointer reasoning for dynamic test generation , 2009, ISSTA.

[14]  Gregg Rothermel,et al.  A Scalable Distributed Concolic Testing Approach: An Empirical Evaluation , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[15]  Bruno Marre,et al.  PathCrawler: Automatic Generation of Path Tests by Combining Static and Dynamic Analysis , 2005, EDCC.

[16]  Koushik Sen,et al.  CUTE: a concolic unit testing engine for C , 2005, ESEC/FSE-13.

[17]  Patrice Godefroid,et al.  Automatic partial loop summarization in dynamic test generation , 2011, ISSTA '11.

[18]  Dawson R. Engler,et al.  KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs , 2008, OSDI.

[19]  Per Runeson,et al.  A survey of unit testing practices , 2006, IEEE Software.

[20]  Sarfraz Khurshid,et al.  Test input generation using dynamic programming , 2012, SIGSOFT FSE.

[21]  George C. Necula,et al.  CIL: Intermediate Language and Tools for Analysis and Transformation of C Programs , 2002, CC.

[22]  Tao Xie,et al.  Generating program inputs for database application testing , 2011, 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011).

[23]  Koushik Sen,et al.  DART: directed automated random testing , 2005, PLDI '05.

[24]  Jian Lu,et al.  Environmental Modeling for Automated Cloud Application Testing , 2012, IEEE Software.

[25]  Moonzoo Kim,et al.  Concolic testing of the multi-sector read operation for flash storage platform software , 2012, Formal Aspects of Computing.

[26]  Klaus Wehrle,et al.  KleeNet: discovering insidious interaction bugs in wireless sensor networks before deployment , 2010, IPSN '10.

[27]  Gogul Balakrishnan,et al.  Feedback-directed unit test generation for C/C++ using concolic execution , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[28]  Frank Tip,et al.  Finding bugs in dynamic web applications , 2008, ISSTA '08.

[29]  Michael D. Ernst,et al.  Feedback-Directed Random Test Generation , 2007, 29th International Conference on Software Engineering (ICSE'07).

[30]  Philippe Herrmann,et al.  Structural Testing of Executables , 2008, 2008 1st International Conference on Software Testing, Verification, and Validation.

[31]  Michael D. Ernst,et al.  Eclat: Automatic Generation and Classification of Test Inputs , 2005, ECOOP.

[32]  Nikolai Tillmann,et al.  Parameterized unit tests , 2005, ESEC/FSE-13.

[33]  Philippe Herrmann,et al.  OSMOSE: automatic structural testing of executables , 2011, Softw. Test. Verification Reliab..

[34]  Nikolai Tillmann,et al.  Retrofitting Unit Tests for Parameterized Unit Testing , 2011, FASE.

[35]  Rupak Majumdar,et al.  Dynamic test input generation for database applications , 2007, ISSTA '07.