An empirical study on constraint optimization techniques for test generation

Constraint solving is a frequent, but expensive operation with symbolic execution to generate tests for a program. To improve the efficiency of test generation using constraint solving, four optimization techniques are usually applied to existing constraint solvers, which are constraint independence, constraint set simplification, constraint caching, and expression rewriting. In this paper, we conducted an empirical study, using these four constraint optimization techniques in a well known test generation tool KLEE with 77 GNU Coreutils applications, to systematically investigate how these optimization techniques affect the efficiency of test generation. The experimental results show that these constraint optimization techniques as well as their combinations cannot improve the efficiency of test generation significantly for ALL-SIZED programs. Moreover, we studied the constraint optimization techniques with respect to two static metrics, lines of code (LOC) and cyclomatic complexity (CC), of programs. The experimental results show that the “constraint set simplification” technique can improve the efficiency of test generation significantly for the programs with high LOC and CC values. The “constraint caching” optimization technique can improve the efficiency of test generation significantly for the programs with low LOC and CC values. Finally, we propose four hybrid optimization strategies and practical guidelines based on different static metrics.创新点我们使用KLEE执行了77个GNU Coreutils程序,用于系统的研究4种流行的约束优化技术如何影响测试用例生成的效率。结果表明,对于所有程序,这些约束优化技术及它们的组合不能大幅提高测试用例生成的效率。 此外,我们研究了程序的两个静态指标(代码行和圈复杂度)跟约束优化技术的关系。结果表明,对于大规模及高圈复杂度程序,使用“约束集简化”技术,对于小规模及低圈复杂度程序,使用“约束缓存”技术,都可以显著提高测试生成效率。最后,基于不同的静态指标,我们提出了四种混合优化策略和指导方针。