Automated coverage calculation and test case generation

This article describes the use of symbolic execution, a formal method of static analysis, to calculate code coverage of a program's existing JUnit test suites. Code coverage is measured with respect to a number of test adequacy criteria, including statement coverage, branch coverage, condition coverage, method coverage, class coverage, and loop coverage. The results of the code coverage calculation is then used to automatically generate JUnit test cases to reach areas of a program that are not sufficiently covered. The level of redundancy of each test case is also calculated during coverage calculation, thereby identifying fully redundant, and partially redundant, test cases. These calculations are packaged as an Eclipse plug-in, named ATCO. ATCO can also be used to perform coverage calculation during a manual execution of a program, allowing testers to measure the effectiveness of manual testing.

[1]  Xavier Lorca,et al.  Choco: an Open Source Java Constraint Programming Library , 2008 .

[2]  Patrick Cousot,et al.  The ASTR ´ EE Analyzer , 2005 .

[3]  Nikolai Tillmann,et al.  DySy: dynamic symbolic execution for invariant inference , 2008, ICSE.

[4]  David Notkin,et al.  Symstra: A Framework for Generating Object-Oriented Unit Tests Using Symbolic Execution , 2005, TACAS.

[5]  B. Aktan,et al.  Improving evolutionary algorithm performance on maximizing functional test coverage of ASICs using adaptation of the fitness criteria , 2002, Proceedings of the 2002 Congress on Evolutionary Computation. CEC'02 (Cat. No.02TH8600).

[6]  James C. King,et al.  A new approach to program testing , 1974, Programming Methodology.

[7]  William G. Griswold,et al.  Dynamically discovering likely program invariants to support program evolution , 1999, Proceedings of the 1999 International Conference on Software Engineering (IEEE Cat. No.99CB37002).

[8]  Laurie Hendren,et al.  Soot: a Java bytecode optimization framework , 2010, CASCON.

[9]  George S. Avrunin,et al.  Using model checking with symbolic execution to verify parallel numerical programs , 2006, ISSTA '06.

[10]  Qian Yang,et al.  A survey of coverage based testing tools , 2006, AST '06.

[11]  Avner Landver,et al.  Coverage-Directed Test Generation Using Symbolic Techniques, Formal Methods in CAD , 1996 .

[12]  H. Amjad,et al.  Combining model checking and theorem proving , 2004 .

[13]  Coen-PorisiniAlberto,et al.  Using symbolic execution for verifying safety-critical systems , 2001 .

[14]  Willem Visser,et al.  Variably interprocedural program analysis for runtime error detection , 2007, ISSTA '07.

[15]  Qian Yang,et al.  A Survey of Coverage-Based Testing Tools , 2009, Comput. J..

[16]  Yannis Smaragdakis,et al.  DSD-Crasher: A hybrid analysis tool for bug finding , 2006, TSEM.

[17]  Klaus Havelund,et al.  Model Checking Programs , 2004, Automated Software Engineering.

[18]  James C. King,et al.  Symbolic execution and program testing , 1976, CACM.

[19]  Patrick Cousot,et al.  The ASTREÉ Analyzer , 2005, ESOP.

[20]  James C. Corbett,et al.  Bandera: extracting finite-state models from Java source code , 2000, ICSE.

[21]  A. Ziv,et al.  Off-The-Shelf Vs . Custom Made Coverage Models , Which Is The One for You ? , 1998 .

[22]  Tao Xie,et al.  Improving Effectiveness of Automated Software Testing in the Absence of Specifications , 2006, 2006 22nd IEEE International Conference on Software Maintenance.

[23]  Mark Lillibridge,et al.  Extended static checking for Java , 2002, PLDI '02.

[24]  Kristina Lundqvist,et al.  Formal Software Verification: Model Checking and Theorem Proving , 2007 .

[25]  Flemming Nielson,et al.  Principles of Program Analysis , 1999, Springer Berlin Heidelberg.

[26]  Yannis Smaragdakis,et al.  JCrasher: an automatic robustness tester for Java , 2004, Softw. Pract. Exp..

[27]  Keijo Heljanko,et al.  Specification coverage aided test selection , 2003, Third International Conference on Application of Concurrency to System Design, 2003. Proceedings..

[28]  Avi Ziv,et al.  User defined coverage—a tool supported methodology for design verification , 1998, DAC.

[29]  Gerard J. Holzmann,et al.  Model-Driven Software Verification , 2004, SPIN.

[30]  David Hovemeyer,et al.  Finding bugs is easy , 2004, SIGP.

[31]  Avner Landver,et al.  Coverage-Directed Test Generation Using Symbolic Techniques , 1996, FMCAD.

[32]  Edmund M. Clarke,et al.  Formal Methods: State of the Art and Future Directions Working Group Members , 1996 .

[33]  Joseph Sifakis,et al.  Model checking , 1996, Handbook of Automated Reasoning.

[34]  Sarfraz Khurshid,et al.  Generalized Symbolic Execution for Model Checking and Testing , 2003, TACAS.

[35]  C. Csallner,et al.  Check 'n' crash: combining static checking and testing , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[36]  Hong Zhu,et al.  Software unit test coverage and adequacy , 1997, ACM Comput. Surv..

[37]  Petri Ihantola.  Test data generation for programming exercises with symbolic execution in Java PathFinder , 2006, Baltic Sea '06.