Test coverage and impact analysis for detecting refactoring faults: a study on the extract method refactoring

Refactoring validation by automated testing is a common practice in agile development processes. However, this practice can be misleading when the test suite is not adequate. Particularly, refactoring faults can be tricky and difficult to detect. While coverage analysis is a standard practice to evaluate a test suite's fault detection capability, there is usually low correlation between coverage and fault detection. In this paper, we present an exploratory study on coverage of refactoring-impacted code, in order to identify shortcomings of test suites, focusing on the Extract Method Refactoring. We consider three open-source projects and their test suites. The results show that, in most cases, the lacking of test case calling the method changed in the refactoring increases the chance of missing faults. Also, a high proportion of test cases that do not cover the callers of that method does not reveal the fault either. Additional analysis of branch coverage on the test cases exercising impacted elements show a higher chance of detecting a fault when branch coverage is also high. It seems reasonable to conclude that a combination of impact analysis with branch coverage could be highly effective in detecting faults introduced by Extract Method.