The Fitness Function for the Job: Search-Based Generation of Test Suites That Detect Real Faults

Search-based test generation, if effective at fault detection, can lower the cost of testing. Such techniques rely on fitness functions to guide the search. Ultimately, such functions represent test goals that approximate — but do not ensure — fault detection. The need to rely on approximations leads to two questions — can fitness functions produce effective tests and, if so, which should be used to generate tests? To answer these questions, we have assessed the fault-detection capabilities of the EvoSuite framework and eight of its fitness functions on 353 real faults from the Defects4J database. Our analysis has found that the strongest indicator of effectiveness is a high level of code coverage. Consequently, the branch coverage fitness function is the most effective. Our findings indicate that fitness functions that thoroughly explore system structure should be used as primary generation objectives — supported by secondary fitness functions that vary the scenarios explored.