Multi-objective Evolutionary Algorithm for String SMT Solver Testing

String Satisfiability Modulo Theories (SMT) solver is widely used in academia and industry. The runtime efficiency of the solvers may have great impact on various software engineering tasks such as automated reasoning and formal verification. Although many studies have been conducted to test SMT solvers, they mainly focus on detecting soundness bugs of SMT solvers. In contrast, only very few studies concentrate on detecting performance defects of SMT solvers. Moreover, in the existing literatures, we observe two major barriers in generating test cases that trigger performance defects for string SMT solvers, i.e., the guidance information barrier and the diversity barrier.In this paper, we propose a multi-objective evolutionary algorithm, MulStringFuzz, to detect performance defects of string SMT solvers. The unique feature of MulStringFuzz lies in the combination of the multi-objective model and the diversity maintenance mechanism. To tackle the guidance information barrier, MulStringFuzz employs multiple objective functions, i.e., the running time, the code coverage, and the test case complexity to guide the test case generation. To tackle the diversity barrier, a tracing based crowding distance mechanism is proposed to ensure the diversity of generated test cases. Extensive experiments are conducted to evaluate the effectiveness of MulStringFuzz, and we investigate how each proposed mechanism contribute to the overall framework. The test cases generated by MulStringFuzz can cover nearly 5,000 more lines of code and trigger 3.25 times performance defects than StringFuzz, which shows that MulStringFuzz can effectively detect performance defect of the String SMT solver.