Investigating the effect of "defect co-fix" on quality assurance resource allocation: A search-based approach

Introducing the concept of "remaining defects" in QA resource prioritization.Introducing "co-fix-awareness" to dynamically rank source code files.Proposing two co-fix-aware ranking algorithms.Empirically comparing several QA resource prioritization algorithms.Investigating the applicability of search algorithms on QA resource prioritization. Allocation of resources to pre-release quality assurance (QA) tasks, such as source code analysis, peer review, and testing, is one of the challenges faced by a software project manager. The goal is to find as many defects as possible with the available QA resources prior to the release. This can be achieved by assigning more resources to the more defect-prone artifacts, e.g., components, classes, and methods. The state-of-the-art QA resource allocation approaches predict the defect-proneness of an artifact using the historical data of different software metrics, e.g., the number of previous defects and the changes in the artifact. Given a QA budget, an allocation technique selects the most defect-prone artifacts, for further investigation by the QA team. While there has been many research efforts on discovering more predictive software metrics and more effective defect prediction algorithms, the cost-effectiveness of the QA resource allocation approaches has always been evaluated by counting the number of defects per selected artifact. The problem with such an evaluation approach is that it ignores the fact that, in practice, fixing a software issue is not bounded to an artifact under investigation. In other words, one may start reviewing a file that is identified as defect-prone and detect a defect, but to fix the defect one may modify not only the defective part of the file under review, but also several other artifacts that are somehow related to the defective code (e.g., a method that calls the defective code). Such co-fixes (fixing several defects together) during analyzing/reviewing/testing of an artifact under investigation will change the number of remaining defects in the other artifacts. Therefore, a QA resource allocation approach is more effective if it prioritizes the artifacts that would lead to the smallest number of remaining defects. Investigating six medium-to-large releases of open source systems (Mylyn, Eclipse, and NetBeans, two releases each), we found that co-fixes happen quite often in software projects (30-42% of the fixes modify more than one artifact). Therefore, in this paper, we first introduce a new cost-effectiveness measure to evaluate QA resource allocation, based on the concept of "remaining defects" per file. We then propose several co-fix-aware prioritization approaches to dynamically optimize the new measure, based on the historical defect co-fixes. The evaluation of these approaches on the six releases shows that (a) co-fix-aware QA prioritization approaches improve the traditional defect prediction-based ones, in terms of density of remaining defects per file and (b) co-fix-aware QA prioritization can potentially benefit from search-based software engineering techniques.