Quest for the Golden Approach: An Experimental Evaluation of Duplicate Crowdtesting Reports Detection

Background: Given the invisibility and unpredictability of distributed crowdtesting processes, there is a large number of duplicate reports, and detecting these duplicate reports is an important task to help save testing effort. Although, many approaches have been proposed to automatically detect the duplicates, the comparison among them and the practical guidelines to adopt these approaches in crowdtesting remain vague. Aims: We aim at conducting the first experimental evaluation of the commonly-used and state-of-the-art approaches for duplicate detection in crowdtesting reports, and exploring which is the golden approach. Method: We begin with a systematic review of approaches for duplicate detection, and select ten state-of-the-art approaches for our experimental evaluation. We conduct duplicate detection with each approach on 414 crowdtesting projects with 59,289 reports collected from one of the largest crowdtesting platforms. Results: Machine learning based approach, i.e., ML-REP, and deep learning based approach, i.e., DL-BiMPM, are the best two approaches for duplicate reports detection in crowdtesting, while the later one is more sensitive to the size of training data and more time-consuming for model training and prediction. Conclusions: This paper provides new insights and guidelines to select appropriate duplicate detection techniques for duplicate crowdtesting reports detection.