1 Duplicate Detection in ProgrammingQuestion Answering Communities

Community based Question Answering (CQA) websites are attracting increasing number of users and contributors in recent years. However, duplicate questions frequently occur in CQA websites and are currently manually identified by the moderators. Automatic duplicate detection, on one hand will alleviate this laborious effort from moderators before taking close actions, on the other hand helps question issuers quickly find answers. A number of studies have looked into related problems, however, very limited works target the duplicate detection in Programming CQA (PCQA), a branch of CQA that is dedicated to programmers. Existing works framed the task as a supervised learning problem on the question pairs and relied on only textual features. Moreover, the issue of selecting candidate duplicates from large volume of historical questions is often un-addressed. To tackle these issues, we model the duplicate detection as a two-stage “ranking-classification” problem over question pairs. In the first stage, we rank the historical questions according to their similarities to the new issued question and select the top ranked ones as candidates to reduce the search space. In the second stage, we develop novel features that capture both textual similarity and latent semantics on question pairs, leveraging techniques in deep learning and information retrieval literature. Experiments on real-world questions about multiple programming languages demonstrate that our method works very well; in some cases up to 25% improvement compared to the state-of-the-art benchmarks.