Software metrics thresholds calculation techniques to predict fault-proneness: An empirical comparison

Abstract Context: Nowadays, fault-proneness prediction is an important field of software engineering. It can be used by developers and testers to prioritize tests. This would allow a better allocation of resources, reducing testing time and costs, and improving the effectiveness of software testing. Non-supervised fault-proneness prediction models, especially thresholds-based models, can easily be automated and give valuable insights to developers and testers on the classification performed. Objective: In this paper, we investigated three thresholds calculation techniques that can be used for fault-proneness prediction: ROC Curves, VARL (Value of an Acceptable Risk Level) and Alves rankings. We compared the performance of these techniques with the performance of four machine learning and two clustering based models. Method: Threshold values were calculated on a total of twelve different public datasets: eleven from the PROMISE Repository and another based on the Eclipse project. Thresholds-based models were then constructed using each thresholds calculation technique investigated. For comparison, results were also computed for supervised machine learning and clustering based models. Inter-dataset experimentation between different systems and versions of a same system was performed. Results: Results show that ROC Curves is the best performing method among the three thresholds calculation methods investigated, closely followed by Alves Rankings. VARL method didn’t give valuable results for most of the datasets investigated and was easily outperformed by the two other methods. Results also show that thresholds-based models using ROC Curves outperformed machine learning and clustering based models. Conclusion: The best of the three thresholds calculation techniques for fault-proneness prediction is ROC Curves, but Alves Rankings is a good choice too. In fact, the advantage of Alves Rankings over ROC Curves technique is that it is completely unsupervised and can therefore give pertinent threshold values when fault data is not available.